Hello everyone.

I'm looking for a way to pass model input data, in terms of sets, parameters and other constants, to a program written in a mainstream language.

The reason is that I have a complex model with lots of parameters of different dimensions and its implementation in a modelling language, precisely AMPL. Then I have to implement a more sophisticated algorithm in a mainstream language, say C++, for the same model, and I need to use, among other things, the parameters of the model in the new implementation

For example, say you have a .dat file like this:

param P:= 1 2 3 4;
param X : A B C :=
1 2 3
4 5 6;

Then I want to be able to pass this file, or something similar, to my C++ program and construct an equivalent data structure, something like

struct myData {int *P, int* X};
myData foo;
// allocate P and X
...
read(file,"P",foo.P); // stores values of param P in foo.P, so foo.P[0]==1 ...
read(file,"X,foo.X);  // stores values of param X in foo.X, so foo.X[0][0]==1, or foo.X[0]["A"]==1

I cannot just read the problem produced by AMPL in lp or other formats because I'm going to use an ad-hoc algorithm and I need to use the raw input data in the code knowing what they represent. The problem printed by AMPL furthermore is "unstructured" as it doesn't keep track of the names and the dimensions of each entity in the model.

The .dat file is, instead, well "structured", i.e. parameters and sets have names, dimensions and the like, and allows to keep track of symbolic names, pretty useful for clearer output.

But there is (IIRC) no parser for AMPL dat files or other formats.

It would come really handy to have a library that exploits the structure of those files. In this way one could read a file by specifying which parameters to read, in a robust way and without describing exactly how the computer has to parse each single token.

So far I've been using a mix of CSV-like parsing. It's ugly for several reasons, the first one being the need to reinvent the wheel every time. I have to decide how to structure the files, how to read them, pay attention not to introduce unwanted white spaces somewhere in the input, etc... All of this while the .dat file sits there unused.

I was suggested using formats like XML, JSON or YAML. Why they have their own merits they can be more verbose, more complex to parse and hard to generate with AMPL and friends. So, AFAIK, they're not worth it.

So, how do you solve this problem? Perhaps I'm missing something?

TIA

Andrea

Edit An example of "structured" data file (notice the names)

param P := 1 2 3;
param T := 1 2;

or even

P: 1 2 3
T: 1 2

An example of usage in an ideal library:

read(file,"P",&P); // stores parameter P values in P array

An example of unstructured data file:

1;2;3;
1;2

To parse the unstructured file you have to break its line into token and so forth.

asked 13 May '14, 12:46

Andrea%20T's gravatar image

Andrea T
384418
accept rate: 0%

edited 13 May '14, 15:43

Ok, @Paul Rubin provided a feasible solution: use a SQL lite (or any other DB-like software) to store model data.

It requires more boilerplate than reading CSV files but, at least, I could forget about manually parsing half-assed files for the next problems.

(13 May '14, 18:25) Andrea T

Rather than trying to generate a structured parameter file from AMPL, it might be easier to look for a file format (SQLite? other SQL?) that can be used as both input to AMPL and input to your program.

link

answered 13 May '14, 15:55

Paul%20Rubin's gravatar image

Paul Rubin ♦♦
14.6k413
accept rate: 19%

edited 13 May '14, 15:58

Well, with some CSV-like parsing I can generate a file and have it parsed in a short time, but it is data-dependent, error-prone and hard to extend as I have to tell the program how to parse each token.

Formats that do not have this problem are YAML, JSON, XML... but are hard to generate from AMPL.

What format do you use to input model data in a C/C++ program?

(13 May '14, 16:06) Andrea T
1

"What format do you use to input model data in a C/C++ program?" (In my case this translates to a Java program; I'm allergic to languages starting with 'C'.) In two of my last three projects, I used undelimited ASCII files (fields separated by spaces, records separated by EOLs) because that's the format inflicted on me by coauthors (who brought the data from earlier work). In the third, I used an ad hoc XML format. If I had it to do again, I would use SQLite files (which I used to record solutions, run times, run parameters).

(13 May '14, 16:26) Paul Rubin ♦♦

Well, Ok, seems DBs are the best option a.t.m..

(13 May '14, 17:42) Andrea T

One question: according to your experience, do you find printing SQL statements with AMPL a good substitute to using AMPL's ODBC handler? I don't usually work on WIndows.

(13 May '14, 18:27) Andrea T

I'm not sure what you mean by "printing SQL statements", and in any case I probably can't answer this. Perhaps someone else can. I moved to Linux several years ago, but have not had the need to connect AMPL to an SQL database since the move. FWIW, AMPL provides details on their ODBC connection on Linux at http://ampl.com/resources/database-and-spreadsheet-table-handlers/#Linux and on connecting to MySQL at https://ampl.github.io/tables/mysql.html.

(14 May '14, 09:49) Paul Rubin ♦♦

Oh, nevermind. I kept ending up on an outdated piece of documentation that said DB handlers were available on Windows only.

thanks!

(14 May '14, 12:51) Andrea T
showing 5 of 6 show 1 more comments

You can use the AMPL API to read the data in the AMPL format. For example:

import java.io.IOException;

import com.ampl.AMPL;
import com.ampl.Set;

public class Test {
  public static void main(String[] args) throws IOException {
    AMPL ampl = new AMPL();
    try {
      ampl.read("test.mod");
      ampl.readData("test.dat");
      Set p = ampl.getSet("P");
      // Compute a sum of elements of the set P.
      double sum = 0;
      for (Object i: p)
        sum += (double)i;
      System.out.format("%g\n", sum);
    } finally {
      ampl.close();
    }
  }
}

test.mod:

set P;
set T;

test.dat:

set P := 1 2 3;
set T := 1 2;

This reads a model and data file and iterates over a set defined in the data file computing the sum of its elements. You can access any data through the API. Currently supported languages are Java and Matlab, other language bindings are on their way.

Note that you can't unambiguously read AMPL data without a model, for example:

param P := 1 2 3 4;

can be interpreted in different ways (P[1, 2, 3] = 4 or P[1] = 2 & P[3] = 4) depending on the declaration of P. However, you can combine model (parameter & set declarations) and data in one file.

link

answered 02 Aug '14, 13:07

vitaut's gravatar image

vitaut
32718
accept rate: 20%

edited 02 Aug '14, 16:29

Resources for reading AMPL intermediate files are here.

link

answered 13 May '14, 13:06

Matthew%20Saltzman's gravatar image

Matthew Salt... ♦
4.7k310
accept rate: 17%

Sorry, unless I'm missing something the .nl files have no "structure". What you get at best is the problem data in terms of variables and constraints. But you can't guess which of those variables or constraints refer to in the original problem. Unless you start counting and keep track of the order in which the solver has written the problem, painful and error-prone.

I'm looking for a way to have the data in the dat file parsed by a mainstream language.

(13 May '14, 13:33) Andrea T
1

Ah, I think I see what you are getting at. One can get a map of names to rows and columns by having AMPL print .row and .col files plus a few other files. See the reference manual Section A.18.4 (http://ampl.com/BOOK/CHAPTERS/24-refman.pdf) Does that help you?

There's also a beta test of a new AMPL API, but it's currently only for Java and Matlab (http://ampl.com/products/apibeta/). That might give access to AMPL's internal problem representation.

(13 May '14, 14:48) Matthew Salt... ♦

I'm sorry, I wasn't clear in the first version of my post. The problem I have doesn't regard rows and columns. It regards data.

Being able to know which variables in a stub refer the variables in the original model would be helpful too, but that's not my problem.

(13 May '14, 15:51) Andrea T
2

Seems to me that all the structure AMPL knows about is captured by the names. It isn't all that easy to recapture the submatrices from name strings, but it is all there.

That said, if you can work in Java or Matlab, you could try the beta-test version of the AMPL API (http://ampl.com/products/apibeta/). Alternatively, the opens-source GLPK includes a parser for GMPL (http://en.wikibooks.org/wiki/GLPK/GMPL_%28MathProg%29), a subset of the AMPL language. Am I getting warmer?

(13 May '14, 16:01) Matthew Salt... ♦

Unless I misread the examples, .row and .columns just contain names of rows and columns. Useful, but not enough to parse correctly data. I see there's a Java API for AMPL. I'll check it later maybe. I'm working with C++ a.t.m. and I'd like something more general than AMPL format so that I can keep using that format and be done with that.

My point is: isn't there an easy-to-parse file format to pass data around? That works well for mainstream languages and modelling languages? Answers seems negative, but thanks anyway!

(13 May '14, 17:41) Andrea T
Your answer
toggle preview

Follow this question

By Email:

Once you sign in you will be able to subscribe for any updates here

By RSS:

Answers

Answers and Comments

Markdown Basics

  • *italic* or _italic_
  • **bold** or __bold__
  • link:[text](http://url.com/ "Title")
  • image?![alt text](/path/img.jpg "Title")
  • numbered list: 1. Foo 2. Bar
  • to add a line break simply add two spaces to where you would like the new line to be.
  • basic HTML tags are also supported

Tags:

×39
×29
×6

Asked: 13 May '14, 12:46

Seen: 2,175 times

Last updated: 02 Aug '14, 16:29

OR-Exchange! Your site for questions, answers, and announcements about operations research.