dat

"Dat is the package manager for datasets. Share files with version control, back up data to servers, browse remote files on demand, and automate long-term data preservation. Secure, distributed, fast.

The Dat Project is the home to open source data sharing applications led by Code for Science & Society, a grant-funded non profit. The Dat Project developed the Decentralized Archive Transport (Dat) protocol, which transfers files in a secure, distributed, and fast network allowing you to focus on the fun work without worrying about moving files around.

Key features

• Secure - Data is encrypted upon transfer and the content is verified on arrival. Prevents third-party access to metadata and content. Learn more.
• Transparent - Changes to data are written in an append-only log, creating a version history that improves transparency and auditability.
• Distributed - With the Dat protocol you'll connect directly to other users or servers sharing or downloading common datasets. Any device can host files to share without the need for centralized servers. Read more.
• Future-proof - Unique links are generated using a public key and thus can be used instantly and forever to verify the dataset from anywhere. You don't need to wait for the entire archive to be hashed before you can begin uploading to peers.
• Fast - Files download from multiple sources. Quickly sync updates by only downloading the new bytes, saving time and bandwidth.

Installation

Visit our site for an installation guide or pick your favorite client application:

• Dat Command Line - You are here! Scroll down for the installation details.
• Dat Desktop - A desktop app to manage multiple Dats on your desktop machine.
• Beaker Browser - An experimental p2p browser with built-in support for the Dat protocol.
• Dat Protocol - Build your own application on the Decentralized Archive Transport (Dat) protocol.
• require('dat') - Node.js library for downloading and sharing Dat archives.

 https://github.com/datproject/dat

https://datproject.org/

pyDatalog

"pyDatalog adds the logic programming paradigm to Python's extensive toolbox, in a pythonic way.  

Logic programmers can now use the extensive standard library of Python, and Python programmers can now express complex algorithms quickly.

Datalog is a truly declarative language derived from Prolog, with strong academic foundations.  Datalog excels at managing complexity.  Datalog programs are shorter than their Python equivalent, and Datalog statements can be specified in any order, as simply as formula in a spreadsheet. 

pyDatalog can be used for:

• simulating intelligent behavior (for games or expert systems), 
• querying complex sets of related information (e.g. in data integration or Natural Language Processing),
• performing recursive algorithms (e.g. on hierarchical data structure)

pyDatalog is derived from previous work by John D. Ramsdell.  It is an open-source project (LGPL) lead by Pierre Carbonnelle (in Belgium).  It is inspired by LogicBlox."

 

https://sites.google.com/site/pydatalog/ 

bcolz

"bcolz provides columnar, chunked data containers that can be compressed either in-memory and on-disk. Column storage allows for efficiently querying tables, as well as for cheap column addition and removal. It is based on NumPy, and uses it as the standard data container to communicate with bcolz objects, but it also comes with support for import/export facilities to/from HDF5/PyTables tables and pandas dataframes.

bcolz objects are compressed by default not only for reducing memory/disk storage, but also to improve I/O speed. The compression process is carried out internally by Blosc, a high-performance, multithreaded meta-compressor that is optimized for binary data (although it works with text data just fine too).

bcolz can also use numexpr internally (it does that by default if it detects numexpr installed) or dask so as to accelerate many vector and query operations (although it can use pure NumPy for doing so too). numexpr/dask can optimize the memory usage and use multithreading for doing the computations, so it is blazing fast. This, in combination with carray/ctable disk-based, compressed containers, can be used for performing out-of-core computations efficiently, but most importantly transparently."

https://github.com/Blosc/bcolz

DAGR

"DAGR is a scalable framework for implementing analysis pipelines using parallel design patterns. DAGR abstracts the pipeline concept into a state machine composed of connected algorithmic units. Each algorithmic unit is written to do a single task resulting in highly modularized, reusable code.

DAGR provides infrastructure for control, communication, and parallelism, you provide the kernels to implement your analyses.
Written in modern C++ and designed to leverage MPI+threading for parallelism, DAGR can leverage the latest HPC hardware including many-core architectures and GPUs. The framework supports a number of parallel design patterns including distributed data, map-reduce, and task based parallelism.

Python bindings expose optimized C++ code to those who prefer the rapid development of Python. In addition DAGR is extensible via C, C++, Fortran, or Python. Algorithms written natively in Python are parallelized over MPI, by adding a single statement to the code."

https://github.com/LBL-EESA/dagr

CliMAF

"CliMAF is an Open Source software, distributed with a GPL-compatible licence. See the licence notice. It is available at CliMAF GitHub repository

The aim of CliMAF is to allow for an actual, easy, collaborative development of climate model outputs assessment suites by climate scientists with varied IT background, and to ultimately share such suites for the benefit of the Climate Science.

So, CliMAF can be described as :

• an aid for handling access to common climate simulations and re-analysis datafiles, and to personnal data files, in a uniform way
• a wrapper which provides you with uniform, handy, combination and caching features around :
  • your own post-processing and analysis scripts and programs
  • other tools sets such as NCO and CDO operators
• a way to share advanced climate diagnostic modules
• an actual repository for such modules
• and a visualisation engine for the corresponding results

 CliMAF is basically a Python-scriptable way to process NetCDF CF compliant climate model outputs which allows:

• to almost forget about accessing input data : you refer to ‘variables’ in ‘simulations’, CliMAF knows a bunch of data organization schemes, you just quote some root locations, usually in configuration files ; [ under development : data can also be on the ESGF ]
• to apply diagnostics (i.e. any post-processing module) coded in any langage, provided they meet very minimal requirements, such as described in section Operators : using external scripts, binaries and python functions ; they can be :
  • either binaries, which accepts command-line arguments, read NetCDF files, and output NetCDF files or graphics (yet only in PNG format)
  • or Python function which accept Masked Arrays data structure as inputs and outputs
• to easily pipe and combine such diagnostic binaries or functions
• to describe the piping using Python scripting, thus building formal expressions in a simple syntax (called CRS for CliMAF Reference Syntax)
• to trigger CRS expression computation only once needed
• to handle a cache of results, which access keys are CRS expressions

A very low-profile knowledge of Python is enough to take full advantage of CliMAF.

See the full, extended table at Contents

• License and acknowledgements
• Requirements and ackowledgements
• Installing, configuring, using
• Examples
• Operators : using external scripts, binaries and python functions
• Standard operators and functions
• Pre-defined projects and datasets
• Available functions
• HowTo...
• Experts’ corner
• Contributing changes for integration in CliMAF
• Whats’ new
• Future steps and wish list
• Community & Contact

http://climaf.readthedocs.io/en/latest/

IUP

"IUP is a multi-platform toolkit for building graphical user interfaces. It offers APIs in three basic languages: C, Lua and LED.

Its library contains about 100 functions for creating and manipulating dialogs.

IUP's purpose is to allow a program to run in different systems without changes - the toolkit provides the application portability. Supported systems include: GTK+, Motif and Windows.

IUP uses an abstract layout model based on the boxes-and-glue paradigm from the T_EX text editor. This model, combined with the dialog-specification language (LED) or with the Lua binding (IupLua) makes the dialog creation task more flexible and independent from the graphics system's resolution.

Currently available interface elements can be categorized as follows:

• Primitives (effective user interaction): dialog, label, button, text, multi-line, list, toggle, canvas, frame, image.
• Composition (ways to show the elements): hbox, vbox, zbox, fill.
• Grouping (definition of a common functionality for a group of elements): radio.
• Menu (related both to menu bars and to pop-up menus): menu, submenu, item, separator.
• Additional (elements built outside the main library): dial, gauge, matrix, tabs, valuator, OpenGL canvas, color chooser, color browser.
• Dialogs (useful predefined dialogs): file selection, message, alarm, data input, list selection.

Hence IUP has some advantages over other interface toolkits available:

• Simplicity: due to the small number of functions and to its attribute mechanism, the learning curve for a new user is often faster.
• Portability: the same functions are implemented in each one of the platforms, thus assuring the interface system's portability.
• Customization: the dialog specification language (LED) and the Lua binding (IupLua) are two mechanisms in which it is possible to customize an application for a specific user with a simple-syntax text file.
• Flexibility: its abstract layout mechanism provides flexibility to dialog creation.
• Extensibility: the programmer can create new interface elements as needed.

IUP is free software, can be used for public and commercial applications."

http://webserver2.tecgraf.puc-rio.br/iup/

https://sourceforge.net/projects/imtoolkit/

https://sourceforge.net/projects/canvasdraw/

FEECa

"FEECa (['fi:ka]) is a library implementing the mathematical framework that the theory of finite element exterior calculus (FEEC), developed by Arnold, Falk and Winther, provides for the discretization of partial differential equations (PDEs) that allows for a universal treatment of a large number of physical problems.

FEECa implements the abstract, mathematical concepts of FEEC and provides a full-fledged basis form generated and framework for computations on differential forms. It handles polynomial differential forms in arbitrary dimensions and implements monomial as well as Bernstein bases for polynomials. The package provides functionality to compute bases of the P_r L^k and P-_r L^k spaces of finite element spaces based on the geometric decomposition proposed by Arnold, Falk and Winther.

Douglas N. Arnold, Richard S. Falk, and Ragnar Winther: Finite element exterior calculus, homological techniques, and applications. J. Acta Numerica. 2006

Douglas N. Arnold, Richard S. Falk, and Ragnar Winther: Geometric decompositions and local bases for spaces of finite element differential forms. J. Computer Methods in Applied Mechanics and Engineering. 2009.
The FEECa online documentation is under construction."

https://github.com/Airini/FEECa