Monday, May 29, 2017

dat

"Dat is the package manager for datasets. Share files with version control, back up data to servers, browse remote files on demand, and automate long-term data preservation. Secure, distributed, fast.

The Dat Project is the home to open source data sharing applications led by Code for Science & Society, a grant-funded non profit. The Dat Project developed the Decentralized Archive Transport (Dat) protocol, which transfers files in a secure, distributed, and fast network allowing you to focus on the fun work without worrying about moving files around.

Key features

  • Secure - Data is encrypted upon transfer and the content is verified on arrival. Prevents third-party access to metadata and content. Learn more.
  • Transparent - Changes to data are written in an append-only log, creating a version history that improves transparency and auditability.
  • Distributed - With the Dat protocol you'll connect directly to other users or servers sharing or downloading common datasets. Any device can host files to share without the need for centralized servers. Read more.
  • Future-proof - Unique links are generated using a public key and thus can be used instantly and forever to verify the dataset from anywhere. You don't need to wait for the entire archive to be hashed before you can begin uploading to peers.
  • Fast - Files download from multiple sources. Quickly sync updates by only downloading the new bytes, saving time and bandwidth.

Installation

Visit our site for an installation guide or pick your favorite client application:
  • Dat Command Line - You are here! Scroll down for the installation details.
  • Dat Desktop - A desktop app to manage multiple Dats on your desktop machine.
  • Beaker Browser - An experimental p2p browser with built-in support for the Dat protocol.
  • Dat Protocol - Build your own application on the Decentralized Archive Transport (Dat) protocol.
  • require('dat') - Node.js library for downloading and sharing Dat archives.
 https://github.com/datproject/dat

https://datproject.org/

pyDatalog

"pyDatalog adds the logic programming paradigm to Python's extensive toolbox, in a pythonic way.  

Logic programmers can now use the extensive standard library of Python, and Python programmers can now express complex algorithms quickly.

Datalog is a truly declarative language derived from Prolog, with strong academic foundations.  Datalog excels at managing complexity.  Datalog programs are shorter than their Python equivalent, and Datalog statements can be specified in any order, as simply as formula in a spreadsheet. 

pyDatalog can be used for:
  • simulating intelligent behavior (for games or expert systems), 
  • querying complex sets of related information (e.g. in data integration or Natural Language Processing),
  • performing recursive algorithms (e.g. on hierarchical data structure)
pyDatalog is derived from previous work by John D. Ramsdell.  It is an open-source project (LGPL) lead by Pierre Carbonnelle (in Belgium).  It is inspired by LogicBlox."
 

Thursday, May 25, 2017

bcolz

"bcolz provides columnar, chunked data containers that can be compressed either in-memory and on-disk. Column storage allows for efficiently querying tables, as well as for cheap column addition and removal. It is based on NumPy, and uses it as the standard data container to communicate with bcolz objects, but it also comes with support for import/export facilities to/from HDF5/PyTables tables and pandas dataframes.

bcolz objects are compressed by default not only for reducing memory/disk storage, but also to improve I/O speed. The compression process is carried out internally by Blosc, a high-performance, multithreaded meta-compressor that is optimized for binary data (although it works with text data just fine too).

bcolz can also use numexpr internally (it does that by default if it detects numexpr installed) or dask so as to accelerate many vector and query operations (although it can use pure NumPy for doing so too). numexpr/dask can optimize the memory usage and use multithreading for doing the computations, so it is blazing fast. This, in combination with carray/ctable disk-based, compressed containers, can be used for performing out-of-core computations efficiently, but most importantly transparently."

https://github.com/Blosc/bcolz

DAGR

"DAGR is a scalable framework for implementing analysis pipelines using parallel design patterns. DAGR abstracts the pipeline concept into a state machine composed of connected algorithmic units. Each algorithmic unit is written to do a single task resulting in highly modularized, reusable code.

DAGR provides infrastructure for control, communication, and parallelism, you provide the kernels to implement your analyses.
Written in modern C++ and designed to leverage MPI+threading for parallelism, DAGR can leverage the latest HPC hardware including many-core architectures and GPUs. The framework supports a number of parallel design patterns including distributed data, map-reduce, and task based parallelism.

Python bindings expose optimized C++ code to those who prefer the rapid development of Python. In addition DAGR is extensible via C, C++, Fortran, or Python. Algorithms written natively in Python are parallelized over MPI, by adding a single statement to the code."

https://github.com/LBL-EESA/dagr

CliMAF

"CliMAF is an Open Source software, distributed with a GPL-compatible licence. See the licence notice. It is available at CliMAF GitHub repository

The aim of CliMAF is to allow for an actual, easy, collaborative development of climate model outputs assessment suites by climate scientists with varied IT background, and to ultimately share such suites for the benefit of the Climate Science.

So, CliMAF can be described as :
  • an aid for handling access to common climate simulations and re-analysis datafiles, and to personnal data files, in a uniform way
  • a wrapper which provides you with uniform, handy, combination and caching features around :
    • your own post-processing and analysis scripts and programs
    • other tools sets such as NCO and CDO operators
  • a way to share advanced climate diagnostic modules
  • an actual repository for such modules
  • and a visualisation engine for the corresponding results
 CliMAF is basically a Python-scriptable way to process NetCDF CF compliant climate model outputs which allows:
  • to almost forget about accessing input data : you refer to ‘variables’ in ‘simulations’, CliMAF knows a bunch of data organization schemes, you just quote some root locations, usually in configuration files ; [ under development : data can also be on the ESGF ]
  • to apply diagnostics (i.e. any post-processing module) coded in any langage, provided they meet very minimal requirements, such as described in section Operators : using external scripts, binaries and python functions ; they can be :
    • either binaries, which accepts command-line arguments, read NetCDF files, and output NetCDF files or graphics (yet only in PNG format)
    • or Python function which accept Masked Arrays data structure as inputs and outputs
  • to easily pipe and combine such diagnostic binaries or functions
  • to describe the piping using Python scripting, thus building formal expressions in a simple syntax (called CRS for CliMAF Reference Syntax)
  • to trigger CRS expression computation only once needed
  • to handle a cache of results, which access keys are CRS expressions
A very low-profile knowledge of Python is enough to take full advantage of CliMAF.

See the full, extended table at Contents

http://climaf.readthedocs.io/en/latest/

IUP

"IUP is a multi-platform toolkit for building graphical user interfaces. It offers APIs in three basic languages: C, Lua and LED.

Its library contains about 100 functions for creating and manipulating dialogs.

IUP's purpose is to allow a program to run in different systems without changes - the toolkit provides the application portability. Supported systems include: GTK+, Motif and Windows.

IUP uses an abstract layout model based on the boxes-and-glue paradigm from the TEX text editor. This model, combined with the dialog-specification language (LED) or with the Lua binding (IupLua) makes the dialog creation task more flexible and independent from the graphics system's resolution.
Currently available interface elements can be categorized as follows:
  • Primitives (effective user interaction): dialog, label, button, text, multi-line, list, toggle, canvas, frame, image.
  • Composition (ways to show the elements): hbox, vbox, zbox, fill.
  • Grouping (definition of a common functionality for a group of elements): radio.
  • Menu (related both to menu bars and to pop-up menus): menu, submenu, item, separator.
  • Additional (elements built outside the main library): dial, gauge, matrix, tabs, valuator, OpenGL canvas, color chooser, color browser.
  • Dialogs (useful predefined dialogs): file selection, message, alarm, data input, list selection.
Hence IUP has some advantages over other interface toolkits available:
  • Simplicity: due to the small number of functions and to its attribute mechanism, the learning curve for a new user is often faster.
  • Portability: the same functions are implemented in each one of the platforms, thus assuring the interface system's portability.
  • Customization: the dialog specification language (LED) and the Lua binding (IupLua) are two mechanisms in which it is possible to customize an application for a specific user with a simple-syntax text file.
  • Flexibility: its abstract layout mechanism provides flexibility to dialog creation.
  • Extensibility: the programmer can create new interface elements as needed.
IUP is free software, can be used for public and commercial applications."

http://webserver2.tecgraf.puc-rio.br/iup/

https://sourceforge.net/projects/imtoolkit/

https://sourceforge.net/projects/canvasdraw/

Wednesday, May 24, 2017

FEECa

"FEECa (['fi:ka]) is a library implementing the mathematical framework that the theory of finite element exterior calculus (FEEC), developed by Arnold, Falk and Winther, provides for the discretization of partial differential equations (PDEs) that allows for a universal treatment of a large number of physical problems.

FEECa implements the abstract, mathematical concepts of FEEC and provides a full-fledged basis form generated and framework for computations on differential forms. It handles polynomial differential forms in arbitrary dimensions and implements monomial as well as Bernstein bases for polynomials. The package provides functionality to compute bases of the P_r L^k and P-_r L^k spaces of finite element spaces based on the geometric decomposition proposed by Arnold, Falk and Winther.

Douglas N. Arnold, Richard S. Falk, and Ragnar Winther: Finite element exterior calculus, homological techniques, and applications. J. Acta Numerica. 2006

Douglas N. Arnold, Richard S. Falk, and Ragnar Winther: Geometric decompositions and local bases for spaces of finite element differential forms. J. Computer Methods in Applied Mechanics and Engineering. 2009.
The FEECa online documentation is under construction."

https://github.com/Airini/FEECa

Tuesday, May 23, 2017

Odespy

"Odespy (ODE Software in Python) offers a unified interface to a large collection of software for solving systems of ordinary differential equations (ODEs). There is also some support for Differential Algebraic Equations (DAEs).

The features include:
  • Pure Python implementations of classical explicit schemes such as the Forward Euler method (also called Euler); Runge-Kutta methods of 2nd, 3rd, and 4th order; Heun's method; Adams-Bashforth methods of 2nd, 3rd, and 4th order; Adams-Bashforth-Moulton methods of 2nd and 3rd order.
  • Pure Python implementations of classical implicit schemes such as Backward Euler; 2-step backward scheme; the theta rule; the Midpoint (or Trapezoidal) method.
  • Pure Python implementations of adaptive explicit Runge-Kutta methods of type Runge-Kutta-Fehlberg of order (4,5), Dormand-Prince of order (4,5), Cash-Karp of order (4,5), Bogacki-Shampine of order (2,3).
  • Wrappers for all FORTRAN solvers in ODEPACK.
  • Wrappers for the wrappers of FORTRAN solvers in scipy: vode and zvode (adaptive Adams or BDF from vode.f); dopri5 (adaptive Dormand-Prince method of order (4,5)); dop853 (adaptive Dormand-Prince method of order 8(5,3)); odeint (adaptive Adams or BDF, basically the same as vode, but in the implementation lsoda from ODEPACK).
  • Wrapper for the Runge-Kutta-Chebyshev formulas of order 2 as offered by the well-known FORTRAN code rkc.f.
  • Wrapper for the Runge-Kutta-Fehlberg method of order (4,5) as provided by the well-known FORTRAN code rkf45.f.
  • Wrapper for the Radau5 method as provided by the well-known FORTRAN code radau5.f. There have been some unidentified problems with running this solver (segmentation fault).
  • Wrapper for some solvers in the odelab.
The ODE problem can always be specified in Python, but for wrappers of FORTRAN codes one can also implement the problem in FORTRAN and avoid callback to Python."

https://github.com/hplgit/odespy

http://hplgit.github.io/odespy/doc/web/index.html

Flask

"Flask is a microframework for Python based on Werkzeug, Jinja 2 and good intentions.

Flask is considered more Pythonic than Django because Flask web application code is in most cases more explicit. Flask is easy to get started with as a beginner because there is little boilerplate code for getting a simple app up and running.

Flask is called a micro framework because it does not require particular tools or libraries.[6] It has no database abstraction layer, form validation, or any other components where pre-existing third-party libraries provide common functions. However, Flask supports extensions that can add application features as if they were implemented in Flask itself. Extensions exist for object-relational mappers, form validation, upload handling, various open authentication technologies and several common framework related tools. Extensions are updated far more regularly than the core Flask program.

The features include:
  • Contains development server and debugger
  • Integrated support for unit testing
  • RESTful request dispatching
  • Uses Jinja2 templating
  • Support for secure cookies (client side sessions)
  • 100% WSGI 1.0 compliant
  • Unicode-based
  • Extensive documentation
  • Google App Engine compatibility
  • Extensions available to enhance features desired
Despite the lack of a major release, Flask has become extremely popular among Python enthusiasts. As of mid 2016, it was the most popular Python web development framework on GitHub."

http://flask.pocoo.org/ 

https://www.fullstackpython.com/flask.html

Using Flask for Scientific Web Applications - http://hplgit.github.io/web4sciapps/doc/pub/web4sa_flask.html


DocOnce

"DocOnce is a modestly tagged (Markdown-like) markup language targeting scientific reports, software documentation, books, blog posts, and slides involving much math and code in the text. From DocOnce source you can generate LaTeX, Sphinx, HTML, IPython notebooks, Markdown, MediaWiki, and other formats. This means that you from a single source can get the most up-to-date publishing technologies for paper, tablets, and phones.

The features include:
  • DocOnce is a modestly tagged markup language (see syntax example), quite like Markdown, but with many more features, aimed at documents with much math and code in the text (see demo).
  • There is extensive support for book projects. In addition to classical LaTeX-based paper books one gets for free fully responsive, modern-looking, HTML-based ebooks for tablets and phones. Parts of books can, e.g., appear in blog posts for discussion and as IPython notebooks for experimentation and annotation.
  • For documents with math and code, you can generate clean plain LaTeX (PDF), HTML (with MathJax and Pygments - embedded in your own templates), Sphinx for attractive web design, Markdown, IPython notebooks, HTML for Google or Wordpress blog posts, and MediaWiki. The LaTeX output has many fancy layouts for typesetting of computer code.
  • DocOnce can also output other formats (though without support for nicely typeset math and code): plain untagged text, Google wiki, Creole wiki, and reStructuredText. From Markdown or reStructuredText you can go to XML, DocBook, epub, OpenOffice/LibreOffice, MS Word, and other formats.
  • The document source is first preprocessed by Preprocess and Mako, which gives you full programming capabilities in the document's text. For example, with Mako it is easy to write a book with all computer code examples in two alternative languages (say Matlab and Python), and you can determine the language at compile time of the document. New user-specific features of DocOnce can also be implemented via Mako.
  • DocOnce extends Sphinx, Markdown, and MediaWiki output such that LaTeX align environments with labels work for systems of equations. DocOnce also adjusts Sphinx and HTML code such that it is possible to refer to equations outside the current web page.
  • DocOnce makes it very easy to write slides with math and code by stripping down running text in a report or book. LaTeX Beamer slides, HTML5 slides (reveal.js, deck.js, dzslides), and Remark (Markdown) slides are supported. Slide elements can be arranged in a grid of cells to easily control the layout.
DocOnce looks similar to Markdown, Pandoc-extended Markdown, and in particular MultiMarkdown. The main advantage of DocOnce is the richer support for writing large documents (books) with much math and code and with tailored output both in HTML and LaTeX. DocOnce also has special support for exercises, quizzes, and admonitions, three very desired features when developing educational material. Books can be composed of many smaller documents that may exist independently of the book, thus lowering the barrier of writing books (see example).

A short scientific report demonstrates the many formats that DocOnce can generate and how mathematics and computer code look like. (Note that at the bottom of the page there is a link to another version of the demo with complete DocOnce commands for producing the different versions.)
Another demo shows how DocOnce can be used to create slides in various formats (HTML5 reveal.js, deck.js, etc., as well as LaTeX Beamer).

DocOnce has support for responsive HTML documents with design and functionality based on Bootstrap styles. A Bootstrap demo illustrates the many possibilities for colors and layouts.

DocOnce also has support for exercises in quiz format. Pure quiz files can be automatically uploaded to Kahoot! online quiz games operated through smart phones (with the aid of quiztools for DocOnce to Kahoot! translation).
Several books (up to over 1000 pages) have been written entirely in DocOnce. The primary format is a publisher-specific LaTeX style, but HTML or Sphinx formats can easily be generated, such as this chapter in Bootstrap style, or the solarized color style as many prefer. Slides can quickly be generated from the raw text in the book. Here are examples in the reveal.js (HTML5) style, or the more traditional LaTeX Beamer style, and even the modern IPython notebook tool, which allows for interactive experimentation and annotation."

https://github.com/hplgit/doconce

http://hplgit.github.io/doconce/doc/pub/tutorial/tutorial.html

https://github.com/hplgit/setup4book-doconce

VACUMM

"VACUMM provides generic and specialized tools for the validation of ocean models, and more especially the MARS model from IFREMER. The heart of VACUMM is a library written mainly in the Python language, whose core can be used for the preprocessing and the postprocessing of oceanic and atmospheric data coming from models or observations. The library for instance also has specialized modules for managing outputs from models and making advanced diagnostics.

Features

  • A huge documentation with a gallery, a lot of examples and the complete API: http://www.ifremer.fr/vacumm
  • Full UV-CDAT support and extensions.
  • Matplotlib graphics with advanced plotting objects like geographical mapping tools.
  • Numerous utilities for manipulating and converting time data.
  • Regridding and interpolation of random or gridded data, in 1D or 2D, with curvilinear grid support.
  • Helper routines for inspecting and reading NetCDF objects in single or multiple file datasets.
  • Generic and specialized 1D and 2D filters working on masked variables.
  • Specialized physical and numerical diagnostics, like dynamics, thermodynamics, spectral analyses, tides, etc.
  • Support and extension of CF conventions for searching or formatting variables.
  • Miscellaneous location utilities such as readers of sigma coordinates for ocean models, or Arakawa grid converters.
  • High level generic interface for reading and post-processing NetCDF data from standard or known dataset, such as model outputs or satellite products.
  • Statistical accumulator for large datasets.
  • Interfaces for working with random and gridded bathymetries, and with shorelines.
  • Utilities for working with masks and selection of spatial data.
  • Utilities for working with input and output remote files.
  • Advanced logging classes.
  • Extensions to sphinx for Fortran and Python.
  • A collection of scripts for some diagnostics.

Dependencies

Mandatory: UVCDAT, configobj.
Optional: seawater, PIL, pytz, paramiko, xlwt, sphinx-fortran, cmocean.

Download

To download VACUMM sources, please go to this page.

http://www.ifremer.fr/vacumm/

https://github.com/VACUMM/sonat

Veros

"Veros, the versatile ocean simulator, is just that: A powerful tool that makes high-performance ocean modeling approachable and fun. Since it is a pure Python module, the days of struggling with complicated model setup workflows, ancient programming environments, and obscure legacy code are finally over.

Veros, the versatile ocean simulator, aims to be the swiss army knife of ocean modeling. It is a full-fledged GCM that supports anything between highly idealized configurations and realistic set-ups, targeting students and seasoned researchers alike. Thanks to its seamless interplay with Bohrium, Veros runs efficiently on your laptop, gaming PC (with experimental GPU support through OpenCL & CUDA), and small cluster. Veros has a clear focus on simplicity, usability, and adaptability - because the Baroque is over.

If you want to learn more about the background and capabilities of Veros, you should check out A short introduction to Veros. If you are already convinced, you can jump right into action, and learn how to get started instead!

Veros supports both a NumPy backend for small-scale problems and a fully parallelized high-performance backend powered by Bohrium using either OpenMP (CPU) or OpenCL (GPU). The underlying numerics are based on pyOM2, an ocean model developed by Carsten Eden (Institut für Meereskunde, Hamburg University).

Veros is currently being developed at Niels Bohr Institute, Copenhagen University.

The features include:
  • a fully staggered 3-D grid geometry (C-grid)
  • support for both idealized and realistic configurations in Cartesian or pseudo-spherical coordinates
  • several friction and advection schemes to choose from
  • isoneutral mixing, eddy-kinetic energy, turbulent kinetic energy, and internal wave energy parameterizations
  • several pre-implemented diagnostics such as energy fluxes, variable time averages, and a vertical overturning stream function (written to netCDF output)
  • pre-configured idealized and realistic set-ups that are ready to run and easy to adapt
  • accessibility, readability, and extensibility - thanks to the power of Python!
http://veros.readthedocs.io/

https://github.com/dionhaefner/veros

Bohrium

"Bohrium provides a runtime environment for efficiently executing vectorized applications using your favourite programming language Python/NumPy, C#, F# on Linux, Windows and MacOS.

Forget handcrafting CUDA/OpenCL, forget threading, mutexes, and locks, use Bohrium!

The features include:
  • Lazy Evaluation, Bohrium will lazy evaluate all Python/NumPy operations until it encounters a “Python Read” such a printing an array or having a if-statement testing the value of an array.
  • Views Bohrium supports NumPy views fully thus operating on array slices does not involve data copying.
  • Loop Fusion, Bohrium uses a fusion algorithm that fuses (or merges) array operations into the same computation kernel that are then JIT-compiled and executed. However, Bohrium can only fuse operations that have some common sized dimension and no horizontal data conflicts.
  • Lazy CPU/GPU Communiction, Bohrium only move data between the host and the GPU when the data is accessed directly by Python or a Python C-extension.
Bohrium implements a new python module bohrium that introduces a new array class bohrium.ndarray which inherent from numpy.ndarray. The two array classes a fully compatible thus one only has to replace numpy.ndarray with bohrium.ndarray in order to utilize the Bohrium runtime system.

The documentation is available at www.bh107.org

https://github.com/bh107/bohrium

Iris

"The Iris library implements a data model to create a data abstraction layer which isolates analysis and visualisation code from data format specifics. The data model we have chosen is the CF Data Model. The implementation of this model we have called an Iris Cube.

Iris currently supports read/write access to a range of data formats, including (CF-)netCDF, GRIB, and PP; fundamental data manipulation operations, such as arithmetic, interpolation, and statistics; and a range of integrated plotting options."

https://github.com/SciTools/iris

ICAR

"ICAR is a simplified atmospheric model designed primarily for climate downscaling, atmospheric sensitivity tests, and hopefully educational uses. At this early stage, the model is still undergoing rapid development, and users are encouraged to get updates frequently.

ICAR combines the wind field from a lower resolution atmospheric model, such as a climate model, with an analytical solution for the effect of topography on that wind field to produce a high-resolution wind field. This wind field is then used to advect heat, moisture, and hydrometeor species (e.g. clouds) around the three dimensional domain, and combines this with microphysical calculations.

To run the model 3D time-varying atmospheric data are required, though an ideal test case can be generated for simple simulations as well. See "Running the Model" below. There are some sample python scripts to help make input forcing files, but the WRF pre-processing system can also be used. Low-resolution WRF output files can be used directly, various reanalysis and GCM output files can be used with minimal pre-processing (just get all the variables in the same netcdf file.) In addition, a high-resolution netCDF topography file is required. This will define the grid that ICAR will run on. Finally an ICAR options file is used to specify various parameters for the model. A sample options file is provided in the run/ directory. 

For an outline of the basic code structure see the ICAR code overview

For reference working with the model code and git, see the ICAR and Git workflow

http://icar.readthedocs.io/en/develop/

https://github.com/NCAR/icar

ParallelO

"A high-level Parallel I/O Library for structured grid applications.

For complete documentation, see our website at http://ncar.github.io/ParallelIO/.

PIO can use NetCDF (version 4.3.3+) and/or PnetCDF (version 1.6.0+) for I/O. Ideally, the NetCDF version should be built with MPI, which requires that it be linked with an MPI-enabled version of HDF5. Optionally, NetCDF can be built with DAP support, which introduces a dependency on CURL. Additionally, HDF5, itself, introduces dependencies on LIBZ and (optionally) SZIP."

https://github.com/NCAR/ParallelIO

KGen

"KGen is a Python tool that extracts partial codes out of a large Fortran application and converts them into a standalone/verifiable/executable kernel.

A kernel is a small software that represents a certain characteristic of a larger application. It can be compiled and run generally without using external library on a single computing node. Due to its simple usage, it can greatly improve productivity of various software engineering tasks such as performance optimization, debugging, porting, verification, and so on.

In addition, a kernel could be an efficient vehcle for enhancing communication between collaborators possibly from various disciplines. For example, a kernel that contains a compiler bug is useful not only for reporting the bug but also for producing and fixing the bug by compiler engineer.

While a kernel is useful for many software engineering tasks, it is generally hard to create one. Mere copying and pasting an interesting block of code generally does not produce compilable software. In manual kernel extraction, it is common to scan through all source files to find required statements such as variable declaration and importing other modules. Furthermore, preparing state data for driving the execution of a generated kernel is generally harder task. For example, if a structured variable contains a pointer variable of another structured variable, user should manually copy those variables, aka, deep copying.

Fortunately, most of kernel extraction task from large Fortran application can be automated through static analysis, which is a core function of KGen.

KGen is simple to use. First user specifies a block of code in a target Fortran software with Linux commands for clean/build/run the software. Using these information, KGen scans through all the necessary source files and marks statements that are required to make the marked block to be a standalone/executable/verifiable kernel. Current version of KGen extensively uses F2PY, an excellant Fortran parser written in Python, to generate Abstract Syntax Trees. KGen traverses the ASTs instead of actually reading text source files for collecting analysis information and eventually selects a minimal set of statements for kernel generation.

KGen also produces input data to drive kernel execution and output data to verify its corrrectness. The data generation supports Fortran derived types that can contain another derived type variables. When KGen-generated kernel is executed, the details of verification result will be automaticaly shown on screen with timing information for performance measurement.

Original use case that we had in mind when we envisioned this tool was performance optimization of large scientific applications. The application we were optimizing has been being actively developed/used by scientists around the world. To keep the pace of fast chainging code, we have to split a large task with many smaller tasks so that several enginners optimize their part of code parallely. Soon after we have this capability of automated kernel generation, it turned out that we could use this technique for other types of tasks including debugging large applications, verification of simulation result, collaborating with compiler vendors, and creating custom benchmark tests. Several kernels that have been extracted using this tool can be downloaded from github.com/NCAR/kernelOptimization github repository."

https://ncar.github.io/kgendocs/

https://github.com/NCAR/KGen

AxiSEM

"AxiSEM is a parallel spectral-element method to solve 3D wave propagation in a sphere with axisymmetric or spherically symmetric visco-elastic, acoustic, anisotropic structures. "

https://github.com/geodynamics/axisem 

Monday, May 22, 2017

pv_atmos

"Python scripting for scientific visualization software ParaView. Historically, pv_atmos has been developed to work with geophysical, and in particular, atmospheric model data (hence the name). However, pv_atmos has evolved into a very general package, and contains routines for visualizing netCDF data, and the capability to show arbitrary axes and labels in a large variety of geometries (linear and logarithmic axes, spherical geometry).

This package is described in an open access peer-reviewed article: Jucker, M 2014. Scientific Visualisation of Atmospheric Data with ParaView. Journal of Open Research Software 2(1):e4, DOI: http://dx.doi.org/10.5334/jors.al. Please cite this work if you use this software for your publications.

The modules and functions within them include

pv_atmos.basic

Provides functionality to read data on a 2D or 3D linear or logarithmic coordinates grid, including time evolution (if present) from a netCDF file. The netCDF should loosely correspond to the Climate and Forecast (FC) conventions. The important attribute is the time coordinate: ParaView will be looking for the "units: xxxx since xxxx" attribute to decide which dimension corresponds to time.

LoadData()
Read a netCDF file, convert linear to logarithmic (e.g. pressure to log-pressure) coordinate if desired, and transform according to prefered aspect ratio.
Cart2Spherical()
Transform rectangular geometry into a sphere with given radius.
Make3D()
Take a dataset with (a) 2D variable(s), and expand the chosen variable as third dimension. Classic example: Ocean bathymetry on a lon-lat grid.
TransformCoords()
If not already done when loading the data, apply coordinate transformation in Cartesian coordinates, according to specified aspect ratio and logarithmic coordinates.
MakeSelectable()
In order to be able to switch any filter's visibility on/off in the GUI's pipeline, call this helper function.
DeleteAll(), HideAll(), ShowAll()
Delete, hide, or show all filters present in the pipeline.
CartWind2Sphere()
Converts zonal and meridional winds (or any velocity) from m/s into degrees longitude per time step and degrees latitude per time step. It can also convert pressure velocity from hPa/s into the new vertical coordinate measure per time step.
Sphere2xyz(), xyz2Sphere()
Convert a given point in spherical (Cartesian) coordinates into Cartesian (spherical) coordinates, given the transformations applied to the data. Helpful to position labels, camera, etc.

pv_atmos.grids

Provides the possibility to add axes, grid lines, planes (cuts), and labels. In case of spherical geometry, one can also add shells, which are spheres of a radius corresponding to a given vertical level. Planes and shells contain data information, and can therefore be used for data analysis as well as grid information.

These routines are not limited to any kind of data, and can be used with any data, or even without data, to add a custom grid to a visualization.

AddGrid()
Add a full grid, including grid lines at custom levels of all dimensions. This includes the appropriate lables of the grid lines, and labeling the axes.
SphericalShells()
Similar to AddGrid() in Cartesian coordinates, this adds shells around a sphere to serve as grid. These shells are labeled with the appropriate level value, and a water mark can be added to the outermost shell.
AddGridPlane()
Add one grid plane along one dimension.
AddGridLabel()
Add one label along one dimension.
AddAxisLabel()
Label one given axis
SphericalLabels()
Label any number of vertical levels in spherical geometry.
WaterMark()
Add a water mark to one of the spherical shells. Nice to brand your viz.
LonLat2Polar()
Project 2D longitude-latitude or 3D longitude-latitude-vertical data onto polar coordinates around the North or South pole. This has a little 3D twist, in that the projection can be domed in the vertical.

https://github.com/mjucker/pv_atmos

ArduBlock

"ArduBlock is a Block Programming Language for Arduino. The language and functions model closely to Arduino Language Reference.

Ardublock is a free graphical programming environment to make programming physical computing with Arduino as easy as drag and drop. It is based on the popular ‘scratch’ programming language, which many will already be familiar with. Ardublock generates code in C, a text based language used by the Arduino IDE, which can then be studied and modified if desired.

Ardublock is an extension to the Arduino IDE so that will need to be downloaded as well (found here https://www.arduino.cc/en/Main/Software)."

https://github.com/stfc/ardublock

FORD

"This is an automatic documentation generator for modern Fortran programs. FORD stands for FORtran Documenter. As you may know, "to ford" refers to crossing a river (or other body of water). It does not, in this context, refer to any company or individual associated with cars.

Ford was written due to Doxygen's poor handling of Fortran and the lack of comparable alternatives. ROBODoc can't actually extract any information from the source code and just about any other automatic documentation software I found was either proprietary, didn't work very well for Fortran, or was limited in terms of how it produced its output. f90doc is quite good and I managed to modify it so that it could handle most of Fortran 2003, but it produces rather ugly documentation, can't provide as many links between different parts of the documentation as I'd like, and is written in Perl (which I'm not that familiar with and which lacks the sort of libraries found in Python for producing HTML content).

The goal of FORD is to be able to reliably produce documentation for modern Fortran software which is informative and nice to look at. The documentation should be easy to write and non-obtrusive within the code. While it will never be as feature-rich as Doxygen, hopefully FORD will be able to provide a good alternative for documenting Fortran projects.

The features are:
  • the ability to extract information about variables, procedures, procedure arguments, derived types, programs, and modules from the source code.
  • the ability to extract documentation from comments in the source code.
  • LaTeX support in documentation using MathJax.
  • searchable documentation, using Tipue Search.
  • author description and social media (including Github!) links.
  • links to download the source code.
  • links to individual files, both in their raw form or in HTML with syntax highlighting.
  • use of Markdown to type-set documentation.
  • links between related parts of the software.
  • Bootstrap CSS for the documentation, making it both functional and pretty.
  • configurable settings.
  • ability to create a hiearchical set of pages containing general information, not associated with any particular part of the source code.
  • display an entry for non-Fortran source files with file-level documentation and syntax highlighted code.
https://github.com/cmacmackin/ford

Scribbler

"Pelican wrapper for scientific note-taking.

Scribbler is a piece of software for managing a notebook from the command line. It was written with scientific note-taking in mind, but could be used for other purposes as well.

The goal was to allow notes to be taken in Markdown, avoiding the need for irritating button pressing when links and images need to be added, with sufficient extensions to make the inclusion of PDF images, LaTeX, BibTeX-style referencing, and various other features possible. Scribbler achieves this by acting as a wrapper for the Pelican static-site generator, for which various additional plugins have been written to give it the desired features. Scribbler also converts the HTML output of Pelican into PDF files which can be printed and placed in a binder along with any hand-written notes.

By default notes are written in Markdown, although they may also be written in reStructuredText or HTML. As Scribbler is just a wrapper for Pelican, it is strongly recommended that you read its documentation on writing content. Note that what they refer to as "articles" are equivalent to notes and what they refer to as "pages" are equivalent to appendices. The Pelican syntax for internal links will work with Scribbler, although Scribbler's directory structure is a little different. Whereas the root of the directory structure seen by Pelican is the content directory, for Scribbler it is the top level of the notebook's directory. However, links will only be generated to the contents of the notes, appendices, and files folders.

In addition to standard Markdown, various plugins are loaded providing extra functionality. Of particular note is Markdown Extra, which provides syntax for such things as footnotes, tables, and more. CodeHilite provides syntax-highlighting for code-snippets, as described in the Pelican documentation. figureAltCaption will convert each image which stands in its own paragraph into a <figure> element, with the alt-text used as a caption.

Additionally, several Pelican plugins are used to provide further functionality. These plugins can be used with reStructuredText and HTML content as well as Markdown.

render_math provides support for LaTeX equations. In Markdown, inline math appears between dollar signs. However, there must be now white spacce before the ending $ (i.e. $x^2$ will render, but $ x^2 $ will not). Math appearing on its own line should use double dollar-signs ($$). \begin{equation} and \end{equation} can also be used and the equation can be labelled and referenced, as in actual LaTeX. Note that this depends on MathJax to work and will only render if there is an internet connection.

The pdf-img plugin, which I wrote specifically for Scribbler, allows PDF, PS, and EPS graphics to be used as the source for images. This plugin will simply scan through all images in each note and appendix. If it ends with the extension pdf, ps, or eps then it will create a PNG thumbnail of the first page of the document. This thumbnail will be inserted as the image source, while the image itself will act as a link to the original PDF/PS/EPS file.

My figure-ref plugin (also written for Scribbler) will look for any figures in the HTML output whose caption begins with the format labelname ::. This will be replaced by a figure number. Any references to {#labelname} in the rest of the note will also be replaced with the correct figure number.

Finally, pelican-cite allows BibTeX-style referencing within your notes. If a bibfile is specified in your notebook settings then it will be used as a database of bibliographic data. This file may, optionally be provided or overridden on a per-note basis by adding the metadata publications_src. Inline references can then be provided with the syntax [@bibtexkey] (for author names with year in parentheses) or [@@bibtexkey] (for author names and year all in parentheses). The inline citation will act as a link to a full bibliography entry at the end of the note."

https://github.com/cmacmackin/scribbler

Pelican

"Static site generator that supports Markdown and reST syntax.

Pelican is a static site generator, written in Python.
  • Write content in reStructuredText or Markdown using your editor of choice
  • Includes a simple command line tool to (re)generate site files
  • Easy to interface with version control systems and web hooks
  • Completely static output is simple to host anywhere
The features are:
  • Chronological content (e.g., articles, blog posts) as well as static pages
  • Integration with external services (e.g., Google Analytics and Disqus)
  • Site themes (created using Jinja2 templates)
  • Publication of articles in multiple languages
  • Generation of Atom and RSS feeds
  • Syntax highlighting via Pygments
  • Importing existing content from WordPress, Dotclear, and other services
  • Fast rebuild times due to content caching and selective output writing
Check out Pelican's documentation for further information."

https://github.com/getpelican/pelican

http://docs.getpelican.com/en/stable/

FOODIE

"Fortran Object-Oriented Differential-equations Integration Environment, FOODIE.

Modern Fortran standards (2003+) have introduced support for Object-Oriented Programming (OOP). Exploiting new features like Abstract Data Type (ADT) is now possible to develop a KISS library providing an awesome environment for the numerical integration of Differential-equations such as Ordinary and Partial Differential Eqautions (ODE, PDE). FOODIE is tailored to the systems arising from the semi-discretization of PDEs, but it is not limited to them.
The FOODIE environment allows the (numerical) solution of general, non linear differential equations system of the form:

IVP

where:
  • U_t = dU/dt;
  • U is the vector of state variables being a function of the time-like independent variable t;
  • R is the (vectorial) residual function, it could be a non linear function of the solution U itself;
  • F is the (vectorial) initial conditions function.
FOODIE is aimed to be a KISS-pure-Fortran library for integrating Ordinary Differential Equations (ODE), it being:
  • Pure Fortran implementation;
  • KISS and user-friendly:
    • simple API, presently based on the Rouson's Abstract Data Type Pattern [8];
    • easy building and porting on heterogeneous architectures;
  • comprehensive solvers set out-of-the-box:
    • explicit schemes:
      • Adams-Bashforth schemes see [7, 12]:
        • 1 step, namely the forward explicit Euler scheme, 1st order accurate;
        • 2 to 16 steps, 2nd to 16th accurate, respectively;
      • Euler (forward explicit) scheme, 1st order accurate;
      • Leapfrog, 2nd order accurate:
        • unfiltered leapfrog, 2nd order accurate, mostly unstable, see [4];
        • Robert-Asselin filtered leapfrog, 1st order accurate, see [4, 5, 6];
        • Robert-Asselin-Williams filtered leapfrog, 3rd order accurate, see [5, 6];
      • Linear Multistep Methods, SSP schemes see [16]:
        • 3 steps, 2nd order accurate;
        • 4 steps, 3rd order accurate;
        • 5 steps, 3rd order accurate;
      • Linear Multistep Methods, SSP with Variable Step Size (VSS) schemes see [17]:
        • 2 steps, 2nd order accurate;
        • 3 steps, 2nd order accurate;
        • 3 steps, 3rd order accurate;
        • 4 steps, 3rd order accurate;
        • 5 steps, 3rd order accurate;
      • Linear Multistep Runge-Kutta Methods SSP schemes see [18]:
        • 2 steps, 2 stages, 3rd order accurate;
        • 3 steps, 2 stages, 3rd order accurate;
        • 4 steps, 3 stages, 5th order accurate;
        • 3 steps, 6 stages, 5th order accurate;
        • 3 steps, 5 stages, 6th order accurate;
        • 3 steps, 7 stages, 7th order accurate;
        • 4 steps, 7 stages, 7th order accurate;
        • 4 steps, 5 stages, 8th order accurate;
        • 5 steps, 9 stages, 8th order accurate;
        • 4 steps, 9 stages, 9th order accurate;
        • 3 steps, 20 stages, 10th order accurate;
      • Runge-Kutta schemes:
        • [+] Linear SSP (of any order) schemes, see [16]:
          • generic s-stages of order (s-1)-th;
          • generic s-stages of order s-th;
        • low-storage schemes, see [1, 2, 3]:
          • 1 stage, namely the forward explicit Euler scheme, 1st order accurate;
          • 2 stages;
          • 3 stages;
          • 4 stages;
          • 5 stages, 4th order accurate, 2N registers, see [3];
          • 6 stages, 4th order accurate, 2N registers, see [9];
          • 7 stages, 4th order accurate, 2N registers, see [9];
          • 12 stages, 4th order accurate, 2N registers, see [10];
          • 13 stages, 4th order accurate, 2N registers, see [10];
          • 14 stages, 4th order accurate, 2N registers, see [10];
        • TVD/SSP schemes, see [1]:
          • 1 stage, namely the forward explicit Euler scheme, 1st order accurate;
          • 2 stages, 2nd order accurate;
          • 3 stages, 3rd order accurate;
          • 4 stages;
          • 5 stages, 4th order accurate;
        • embedded (adaptive) schemes:
          • Heun-Euler, 2 stages, 2nd order accurate;
          • Runge-Kutta-Fehlberg, 5 stages, 4th order accurate;
          • Runge-Kutta-Cash-Karp, 6 stages, 5th order accurate, see [13];
          • Prince-Dormand, 7 stages, 4th order accurate, see [11];
          • Calvo, 9 stages, 6th order accurate, see [14];
          • Feagin, 17 stages, 10th order accurate, see [15];
    • implicit schemes:
      • Runge-Kutta schemes;
      • Adams-Moulton schemes:
        • 0 step, 1st order accurate;
        • 1 step, 2nd accurate;
        • 2 steps, 3rd accurate;
        • 3 steps, 4th accurate;
      • Backward Differentiation Formula schemes:
        • 1 step, namely the backward implicit Euler scheme, 1st order accurate;
        • 2 to 6 steps, 2nd to 6th accurate, respectively;
    • predictor-corrector schemes:
      • Adams-Bashforth-Moulton schemes:
        • 1 step, AB(1)-AM(0), 1st order accurate;
        • 2 steps, AB(2)-AM(1), 2nd accurate;
        • 3 steps, AB(3)-AM(2), 3rd accurate;
        • 4 steps, AB(4)-AM(3), 4th accurate;
  • efficient and non intrusive:
    • FOODIE environment is unaware of any eventual parallel paradigms the clients used, but it is proved to preserve high scalability on parallel architectures such as:
      • OpenMP directive-based codes on shared memory multi/many cores architectures;
      • CoArray Fortran (CAF) based codes for Partitioned Global Address Space (PGAS) programming model;
      • MPI based code on distributed memory clusters;
      • GPGPU/accelerators device enabled codes;
  • Tests-Driven Developed (TDD):
  • well documented:
  • collaborative developed on GitHub;
  • FOSS licensed
https://github.com/Fortran-FOSS-Programmers/FOODIE

FLAP

"Fortran command Line Arguments Parser for poor people.

Modern Fortran standards (2003+) have introduced support for Command Line Arguments (CLA), thus it is possible to construct nice and effective Command Line Interfaces (CLI). FLAP is a small library designed to simplify the (repetitive) construction of complicated CLI in pure Fortran (standard 2003+). FLAP has been inspired by the python module argparse trying to mimic it. Once you have defined the arguments that are required by means of a user-friendly method of the CLI, FLAP will parse the CLAs for you. It is worthy of note that FLAP, as argparse, also automatically generates help and usage messages and issues errors when users give the program invalid arguments.

The features:
  • User-friendly methods for building flexible and effective Command Line Interfaces (CLI);
  • comprehensive Command Line Arguments (CLA) support:
    • support optional and non optional CLA;
    • support boolean CLA;
    • support positional CLA;
    • support list of allowable values for defined CLA with automatic consistency check;
    • support multiple valued (list of values, aka list-valued) CLA:
      • compiletime sized list, e.g. nargs='3';
      • runtime sized list with at least 1 value, e.g. nargs='+';
      • runtime sized list with any size, even empty, e.g. nargs='*';
    • support mutually exclusive CLAs;
    • self-consistency-check of CLA definition;
    • support fake CLAs input from a string;
    • support fake CLAs input from environment variables;
  • comprehensive command (group of CLAs) support:
    • support nested subcommands;
    • support mutually exclusive commands;
    • self-consistency-check of command definition;
  • automatic generation of help and usage messages;
  • consistency-check of whole CLI definition;
  • errors trapping for invalid CLI usage;
  • POSIX style compliant;
  • automatic generation of MAN PAGE using your CLI definition!;
  • replicate all the useful features of argparse;
  • implement docopt features.
  • implement click features.
https://github.com/szaghi/FLAP

PreForM

"A very simple and stupid preprocessor for modern Fortran projects.

PreForM.py supports the most used cpp pre-processing directives and provides advanced features typical of templating systems. Even if PreForM.py is currently Fortran-agnostic (it being usable within any programming languages) it is focused on Fortran programming language.

The features are:

  • Easy-extensible: PreForM.py is just a less-than 500 lines of Python statements... no bad for a poor-cpp-preprocessor improvement;


  • well integrated with a powerful yet simple automatic Building System for Fortran poor-men, namely FoBiS.py;
  • simple command line interface;
  • support the most used cpp pre-processing directives:
    • conditionals:
      • operators (also nested):
        • defined MACRO or defined(MACRO);
        • EXPRESSION || EXPRESSION (logic or);
        • EXPRESSION && EXPRESSION (logic and);
      • #if EXPRESSION;
      • #elif EXPRESSION;
      • #ifdef MACRO;
      • #ifndef MACRO;
      • #else;
      • #endif;
    • macros:
      • standard predefined macros:
        • __FILE__;
        • __LINE__;
        • __DATE__;
        • __TIME__;
      • expansion;
      • stringification;
      • concatenation;
      • variadic macros;
      • object-like macros:
        • #define MACRO [VALUE], VALUE is optional;
      • function-like macros:
        • #define FUNCTION FUNCTION_DEFINITION;
      • #undef;
    • #include;
  • pythonic Template System:
    • #PFM for EXPRESSION-#PFM endfor pairs loop control (only for one iteration counter at time
    Note that in general the cpp pre-processing directives should start at the first column, the symbol # being the first one. PreForM.py relaxes this rule allowing any number of leading white spaces before #.

    https://github.com/szaghi/PreForM 
  • FoBiS

    "A KISS tool for automatic building modern Fortran projects.

    The features:
    • :zap: Automatic parsing of files for dependency-hierarchy creation in case of use and include statements;
    • :zap: automatic building of all programs found into the root directory parsed or only a specific selected target;
    • avoid unnecessary re-compilation (algorithm based on file-timestamp value);
    • simple command line interface (CLI);
    • friendly support for external libraries linking:
      • direct linking with full libraries paths;
      • relative linking via extending linker search paths;
    • support for widely used compilers:
      • GNU Fortran Compiler;
      • Intel Fortran Compiler;
      • g95 Fortran Compiler;
      • IBM XL Fortran Compiler;
      • NAG Fortran Compiler;
      • PGI Fortran Compiler;
    • custom compiler support;
    • :zap: configuration-files-free;
    • ... but also configuration-file driven building for complex buildings;
    • :zap: parallel compiling enabled by means of concurrent multiprocessing jobs;
    • :zap: support submodules;
    • advanced automatic (re-)building algorithms:
      • :zap: automatic (re-)building when compiling flags change with respect the previous building;
      • :zap: automatic (re-)building when linked library change;
      • :zap: automatic building of projects upon which the actual project depends;
    • :zap: advanced automatic micro-atomic introspective unittest support by means of doctests;
    • :zap: generation of GNU Make makefile with rules fully supporting dependency-hierarchy for make-irreducible users;
    • easy-extensible;
    • well integrate with a flexible pythonic pre-processor, PreForM.py.
    https://github.com/szaghi/FoBiS