Big data is (amongst other principles) about working with data for which the dataset is so big, that it cannot fit on one computer node, that it cannot fit into a relational database. This means the data needs to be stored in distributed systems like Infinispan, Hadoop, ... A good example of big data is twitter's firehose or google maps / openstreet maps.

Are there any well-described optimization problems for big data, with datasets and results available? Any challenges or contests? Any good, readable research papers?

asked 03 Jul '13, 05:07

Geoffrey%20De%20Smet's gravatar image

Geoffrey De ... ♦
3.6k42766
accept rate: 6%

edited 03 Jul '13, 08:21


link

answered 04 Jul '13, 03:10

jfpuget's gravatar image

jfpuget
2.5k310
accept rate: 8%

Big Data isn't just large volume of data. It is also about non structured data, or data in motion, see Big Data for dummies

We're having some big data optimization projects at IBM, but I can't disclose them for now unfortunately.

link

answered 03 Jul '13, 07:27

jfpuget's gravatar image

jfpuget
2.5k310
accept rate: 8%

1

Too bad that nothing of that is open. I am thinking of doing a proof of concept (open source of course :) of hooking OptaPlanner to Infinispan. If any else is implementing the same use case, it would make for an interesting comparison.

(03 Jul '13, 08:29) Geoffrey De ... ♦
Your answer
toggle preview

Follow this question

By Email:

Once you sign in you will be able to subscribe for any updates here

By RSS:

Answers

Answers and Comments

Markdown Basics

  • *italic* or _italic_
  • **bold** or __bold__
  • link:[text](http://url.com/ "Title")
  • image?![alt text](/path/img.jpg "Title")
  • numbered list: 1. Foo 2. Bar
  • to add a line break simply add two spaces to where you would like the new line to be.
  • basic HTML tags are also supported

Tags:

×3

Asked: 03 Jul '13, 05:07

Seen: 1,239 times

Last updated: 04 Jul '13, 03:10

OR-Exchange! Your site for questions, answers, and announcements about operations research.