Monday, May 7, 2012

Code for Single, Double and Triple Exponential Forecasting

Recently I got interested in analyzing trends in time series and explored few forecasting techniques.

A good read to start are :
  • http://www.itl.nist.gov/div898/handbook/pmc/section4/pmc4.htm
  • http://en.wikipedia.org/wiki/Moving_average
  • http://en.wikipedia.org/wiki/Exponential_smoothing
First thing I did was to plot SMA and EMA over existing data with varying windows sizes and alpha values. This gives a initial idea of how your time series data is fluctuating with time. I used a awesome tool FLOT to visualize. I didn't get time to explore awesome library, but you may try http://www.r-project.org/

Next step after visualizing, is to find trends and forecast. Before forecasting future data points, I suggest forecast over current data set and try to get close fit as possible. Some of my observations from studying all the 3 smoothing methods are :
1) Single exponential should be used to when there are hardly any trends. Forecast value is almost equivalent to last data point. Only gain you get out of this method is to gain insight to alpha value by minimizing Mean Squared Error.
2) Double exponential method gives you better forecast values when there is some trend...say continuously going up or down. Forecasts are much better than single exponential. Playing with gamma value is fun.
3) Triple exponential method is good with trend plus season. By carefully iterating over period,alpha,beta,gamma, and each time minimizing MSE, one can get pretty close forecasts.

Here are my codes used :

 /**
  * http://www.itl.nist.gov/div898/handbook/pmc/section4/pmc431.htm
  * http://www.itl.nist.gov/div898/handbook/pmc/section4/pmc432.htm
  * @param data - input data
  * @param alpha - good value between 0.1-0.9
  * @param numForecasts - ahead forecasts
  * @return
  */
 public static double[] singleExponentialForecast(double[] data, double alpha, int numForecasts) {
  double[] y = new double[data.length + numForecasts];
  y[0] = 0;
  y[1] = data[0];
  int i = 2;
  for (i = 2; i < data.length; i++) {
   y[i] = alpha * data[i - 1] + (1 - alpha) * y[i - 1];
  }

  for (int j = 0; j < numForecasts; j++, i++) {
   y[i] = alpha * data[data.length - 1] + (1 - alpha) * y[i - 1];
  }
  return y;
 }

 /**
  * http://www.itl.nist.gov/div898/handbook/pmc/section4/pmc433.htm
  * http://www.itl.nist.gov/div898/handbook/pmc/section4/pmc434.htm
  * @param data
  * @param alpha
  * @param gamma
  * @param initializationMethod
  * @param numForecasts
  * @return
  */
 public static double[] doubleExponentialForecast(double[] data, double alpha, double gamma, int initializationMethod, int numForecasts) {
  double[] y = new double[data.length + numForecasts];
  double[] s = new double[data.length];
  double[] b = new double[data.length];
  s[0] = y[0] = data[0];
  
  if(initializationMethod==0) {
   b[0] = data[1]-data[0];
  } else if(initializationMethod==1 && data.length>4) {
   b[0] = (data[3] - data[0]) / 3;
  } else if(initializationMethod==2) {
   b[0] = (data[data.length - 1] - data[0])/(data.length - 1);
  }
  
  int i = 1;
  y[1] = s[0] + b[0];
  for (i = 1; i < data.length; i++) {
   s[i] = alpha * data[i] + (1 - alpha) * (s[i - 1]+b[i - 1]);
   b[i] = gamma * (s[i] - s[i - 1]) + (1-gamma) * b[i-1];
   y[i+1] = s[i] + b[i];
  }

  for (int j = 0; j < numForecasts ; j++, i++) {
   y[i] = s[data.length-1] + (j+1) * b[data.length-1];
  }
  
  return y;
 }
 
 public static double TSAError(double[] data, double[] forecast) {
  double mad = 0.0;
  double mse = 0.0;
  double diff = 0.0;

  for (int i = 0; i < data.length; i++) {
   diff = data[i] - forecast[i];
   mad += Math.abs(diff);
   mse += Math.pow(Math.abs(diff), 2.0);
  }
  
  return mse/data.length;
 }

Triple Exponential code can be found at http://n-chandra.blogspot.in/2011/04/holt-winters-triple-exponential.html
PS: Let me know if code has problems

Monday, April 9, 2012

How to add pgp public keys for mongo respository

While installing mongo db, if you are trying to get pgp key using
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv 7F0CEB10

you might receive following error :

Executing: gpg --ignore-time-conflict --no-options --no-default-keyring --secret-keyring /etc/apt/secring.gpg --trustdb-name /etc/apt/trustdb.gpg --keyring /etc/apt/trusted.gpg --primary-keyring /etc/apt/trusted.gpg --keyserver keyserver.ubuntu.com --recv 7F0CEB10
gpg: requesting key 7F0CEB10 from hkp server keyserver.ubuntu.com
gpgkeys: HTTP fetch error 7: couldn't connect to host
gpg: no valid OpenPGP data found.
gpg: Total number processed: 0

You can fix this by storing pgp public key written in http://www.mongodb.org/display/DOCS/Ubuntu+and+Debian+packages into a file say mongo.key and then execute "sudo apt-key add mongo.key" and then proceed with "sudo apt-get update"

Saturday, April 7, 2012

Store ajax json response into javascript variable using jquery

If you wish to make an ajax call, and assign response json into a variable directly...here is a small hack :
function getJson(url) {
 return JSON.parse($.ajax({
     type: 'GET',
     url: url,
     dataType: 'json',
     global: false,
     async:false,
     success: function(data) {
         return data;
     }
 }).responseText);
}

var myJsonObj = getJson('myjsonurl');

Wednesday, March 21, 2012

Umarshalling JSON and XML

If you want to unmarshall json or xml inputstream to Java object, here are the functions :
public static <T> T unmarshalXML(InputStream is, Class<T> c)
   throws JAXBException {
  JAXBContext jc = JAXBContext.newInstance(c);
  Unmarshaller u = jc.createUnmarshaller();
  T response = (T) u.unmarshal(is);
  return response;
 }

 public static <T> T unmarshalJSON(InputStream is, Class<T> c)
   throws JAXBException, IOException, JSONException, XMLStreamException {
  JAXBContext jc = JAXBContext.newInstance(c);
  Unmarshaller u = jc.createUnmarshaller();
  String sJson = IOUtils.toString(is);
  JSONObject obj = new JSONObject(sJson);
  Configuration config = new Configuration();
  MappedNamespaceConvention con = new MappedNamespaceConvention(config);
  XMLStreamReader xmlStreamReader = new MappedXMLStreamReader(obj, con);
  T response = (T) u.unmarshal(xmlStreamReader);
  return response;
 }

Friday, March 16, 2012

Installing ReviewBoard plugin in Eclipse IDE


ReviewBoard (www.reviewboard.org/) is a nice code review web tool. ereviewboard (http://marketplace.eclipse.org/content/ereviewboard) is the corresponding plugin for Eclipse IDE. Below are the steps mentioned to integrate it with Eclipse IDE :

  • Goto Help->Install New software
  • Use URL http://rombert.github.com/ereviewboard/update/ and install "Mylyn Reviews Connector: ReviewBoard" and "Mylyn Reviews Connector: Review Board Subeclipse integration" (there are 3 connectors available, choose the appropriate one...here i am assuming subversion)
  • If subeclipse is not installed, use url http://subclipse.tigris.org/update_1.6.x
  • After installation, Open the task repositories view by navigating to Window -> Show View -> Other -> Mylyn -> Task Repositories
  • Click the "Add Task Repository" button located in the view's toolbar.
  • Select reviewboard and enter server as your reviewboard weburl (ex : http://192.168.x.x), username, password (Save password) and finish.
  • Now, Right-click on a Project and select Team -> Create Review Request will post to reviewboard (if you dont see such a option trying restarting eclipse)

Some helpful links :
http://help.eclipse.org/galileo/index.jsp?topic=/org.eclipse.mylyn.help.ui/userguide/Task-Repositories.html
https://github.com/rombert/ereviewboard/wiki/Subclipse-integration

Remarks :
This tool can only post diff's of 1 project to reviewboard. Multiple project diff are not supported

Listing all entities in a JPA

Sometimes it may be a usecase scenario to find whether a particular class is an entity managed by persistence context. If you have entityManager or entityManagerFactory you can easily do that :
 Metamodel meta = entityManagerFactory.getMetamodel();
 // or
 Metamodel meta = entityManager.getEntityManagerFactory().getMetamodel();

 // to iterate over all classes
 for (EntityType<?> e : meta.getEntities()) {
  // get entity class
  Class c = e.getJavaType();
  // get entity name as string
  String entityName = e.getName(); //or c.getName()
 }

 // test a particular class is entity
 // will throw java.lang.IllegalArgumentException if not an entity
 meta.entity(inputClass);

Curious Case in MYSQL : Lock wait timeout exceeded on INSERT

Sounds strange, that how can a insert be locked or timed out . I had a innodb table with very frequent inserts, updated and deletes. After every few minutes, one of the inserts got timed out (50 sec is default value for innodb_lock_wait_timeout). My understanding was that timeouts happen when some other thread/transaction holds a exclusive record lock (select .. from update) for a long time. So how can a non-existent new row be already locked. I do not have a proper answer to this.

What solved my problem was dropping index and foreign key mapping, which were luckily irrelevant. Random guess is that innodb locks a range of index on insert. If you have an answer do let me know !