Category: Tech

Google App Engine WTFs

“Bad argument: Multiple entries with same key”

If you get the error "Bad argument: Multiple entries with same key" when you’re trying to get modules working in Google App Engine, it’s because you’ve either:

  1. Forgotten to include a <module> tag in more than one of your appengine-web.xml files, meaning that now you (implicitly) have more than one module named “default”.
  2. Named more than one of your modules the same name (probably a copy / paste mistake after duplicating a WAR folder).

Unfortunately, the error message from AppCfg doesn’t help at all to realize this.

ClassNotFoundException

Another annoying error is the ClassNotFoundException when you forget to add a package statement to your Servlet source code file:

W 2014-03-26 19:26:23.632 EXCEPTION java.lang.ClassNotFoundException: com.test.SomeServlet at com.google.appengine.runtime.Request.process-77d3977eb12bfdcf(Reques
E 2014-03-26 19:26:23.634 javax.servlet.ServletContext log: unavailable javax.servlet.UnavailableException: com.test.SomeServlet at org.mortbay.jetty.servlet.Holde
W 2014-03-26 19:26:23.709 Failed startup of context com.google.apphosting.utils.jetty.RuntimeAppEngineWebAppContext@61b097{/,/base/data/home/apps/s~sometest/someservlet:1.
C 2014-03-26 19:26:23.713 Uncaught exception from servlet javax.servlet.UnavailableException: Initialization failed. at com.google.apphosting.runtime.jetty.AppVersionHandlerMa
I 2014-03-26 19:26:23.715 This request caused a new process to be started for your application, and thus caused your application code to be loaded for the first time. This requ
E 2014-03-26 19:26:23.715 Process terminated because it failed to respond to the start request with an HTTP status code of 200-299 or 404.

Failure to do so means that the CLASSPATH will be correct, but your class idiotically won’t be found even though its path in the .ear file will appear to be correct. This should be easily detected by any decent IDE, but in my case I didn’t pick up the mistake as I was using emacs and mvn directly, which didn’t have the smarts to know what was going on.

Update 1: “WARNING: /_ah/start: javax.servlet.UnavailableException: java.lang.IllegalAccessException: Class org.mortbay.jetty.servlet.Holder can not access a member of class”

It means that the Servlet class you defined for a particular module is not “public”. Oops.

400 Bad Request: Invalid runtime or the current user is not authorized to use it.

You may tear out your hair if you see the following messages. Make sure to do mvn install before doing mvn appengine:update in the EAR folder. This kind of bullshit makes me hate Maven.

Grails, Groovy, Scala, and Akka

The interop between these software components is somewhat unexplored territory, but they seem to work alright together. Once you figure out what the Grails conventions are for calling into Scala and where to put the Akka application.conf file (right next to the application.properties file in the same directory), everything starts up just fine.

The Grails dependency injection mechanism for getting references to Services doesn’t work with the way of doing things I’m about to describe. I may need to revisit this in the future, if the convenience of having that becomes greater than keeping everything actor-related in Scala. I have no desire to wrap Akka Actors in Groovy and will eventually need to create a common Java interface that Scala can use to call into the Groovy side, and I will need to pass a reference for that to the Scala instance. But that comes later.

To get the various pieces working together, the Akka dependencies and the Scala plugin for Grails need to be added to the BuildConfig.groovy:

BootStrap.groovy looks something like:

AkkaClusterService.scala looks something like:

application.conf looks something like:

Now as long as you have a seed node in your cluster application listening on 127.0.0.1:2551, the cluster setup should work just fine when you fire off grails run-app.

Starting AkkaClusterService.
[INFO] [03/25/2014 12:10:55.964] [localhost-startStop-1] [Remoting] Starting remoting
[INFO] [03/25/2014 12:10:56.379] [localhost-startStop-1] [Remoting] Remoting started; listening on addresses :[akka.tcp://AkkaCluster@127.0.0.1:53884]
[INFO] [03/25/2014 12:10:56.412] [localhost-startStop-1] [Cluster(akka://AkkaCluster)] Cluster Node [akka.tcp://AkkaCluster@127.0.0.1:53884] - Starting up...
[INFO] [03/25/2014 12:10:56.502] [localhost-startStop-1] [Cluster(akka://AkkaCluster)] Cluster Node [akka.tcp://AkkaCluster@127.0.0.1:53884] - Registered cluster JMX MBean [akka:type=Cluster]
[INFO] [03/25/2014 12:10:56.502] [localhost-startStop-1] [Cluster(akka://AkkaCluster)] Cluster Node [akka.tcp://AkkaCluster@127.0.0.1:53884] - Started up successfully
[INFO] [03/25/2014 12:10:56.551] [AkkaCluster-akka.actor.default-dispatcher-15] [Cluster(akka://AkkaCluster)] Cluster Node [akka.tcp://AkkaCluster@127.0.0.1:53884] - Metrics will be retreived from MBeans, and may be incorrect on some platforms. To increase metric accuracy add the 'sigar.jar' to the classpath and the appropriate platform-specific native libary to 'java.library.path'. Reason: java.lang.ClassNotFoundException: org.hyperic.sigar.Sigar
[INFO] [03/25/2014 12:10:56.558] [AkkaCluster-akka.actor.default-dispatcher-15] [Cluster(akka://AkkaCluster)] Cluster Node [akka.tcp://AkkaCluster@127.0.0.1:53884] - Metrics collection has started successfully
| Server running. Browse to http://localhost:8080/grailshelloworld
| Application loaded in interactive mode. Type 'stop-app' to shutdown.
| Enter a script name to run. Use TAB for completion:
....

If it worked, you’ll see something like:

[INFO] [03/25/2014 12:51:40.910] [AkkaCluster-akka.actor.default-dispatcher-16] [Cluster(akka://AkkaCluster)] Cluster Node [akka.tcp://AkkaCluster@127.0.0.1:56797] - Welcome from [akka.tcp://AkkaCluster@127.0.0.1:2551]
[INFO] [03/25/2014 12:51:40.948] [AkkaCluster-akka.actor.default-dispatcher-3] [akka.tcp://AkkaCluster@127.0.0.1:56797/user/$a] Member is Up: akka.tcp://AkkaCluster@127.0.0.1:2551
[INFO] [03/25/2014 12:51:41.108] [AkkaCluster-akka.actor.default-dispatcher-14] [akka.tcp://AkkaCluster@127.0.0.1:56797/user/$a] Member is Up: akka.tcp://AkkaCluster@127.0.0.1:56797

If it didn’t work, you’ll see something like:

[WARN] [03/25/2014 12:10:57.912] [AkkaCluster-akka.remote.default-remote-dispatcher-7] [akka.tcp://AkkaCluster@127.0.0.1:53884/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FAkkaCluster%40127.0.0.1%3A2551-0] Association with remote system [akka.tcp://AkkaCluster@127.0.0.1:2551] has failed, address is now gated for [5000] ms. Reason is: [Association failed with [akka.tcp://AkkaCluster@127.0.0.1:2551]].
[INFO] [03/25/2014 12:10:57.929] [AkkaCluster-akka.actor.default-dispatcher-5] [akka://AkkaCluster/deadLetters] Message [akka.cluster.InternalClusterAction$InitJoin$] from Actor[akka://AkkaCluster/system/cluster/core/daemon/joinSeedNodeProcess#877024691] to Actor[akka://AkkaCluster/deadLetters] was not delivered. [1] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[INFO] [03/25/2014 12:10:57.929] [AkkaCluster-akka.actor.default-dispatcher-5] [akka://AkkaCluster/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FAkkaCluster%40127.0.0.1%3A2551-0/endpointWriter] Message [akka.actor.FSM$Timer] from Actor[akka://AkkaCluster/deadLetters] to Actor[akka://AkkaCluster/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FAkkaCluster%40127.0.0.1%3A2551-0/endpointWriter#1345637125] was not delivered. [2] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.

That’s it. It should now be possible to define case classes in Scala and to pass them along w/o any need to write Groovy adapters. Or, if you’re feeling adventurous, write Groovy classes using the @Scalify annotation to pass to Scala.

Wash, rinse, and repeat.

Setting up Groovy Environment Manager on Windows

If you have msysgit set up on your Windows machine, then you have the two the requirements to install GVM: bash and curl. Unfortunately, the MinGW environment provided by Git doesn’t completely work. GVM installs fine, but when you go to install Groovy or Grails, you get an error like:

$ gvm install groovy

Downloading: groovy 2.2.2

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100 28.3M  100 28.3M    0     0  2861k      0  0:00:10  0:00:10 --:--:-- 3033k

Installing: groovy 2.2.2
Done installing!

Do you want groovy 2.2.2 to be set as default? (Y/n): y

Setting groovy 2.2.2 as default.
ln: creating symbolic link /c/Users/Test/.gvm/groovy/current' to /c/Users/Test/.gvm/groovy/2.2.2': Permission denied

And for Grails, it’s the same:

Setting grails 2.3.7 as default.
ln: creating symbolic link /c/Users/Test/.gvm/grails/current' to /c/Users/Test/.gvm/grails/2.3.7': Permission denied

The files are in place, they just need to be properly symlinked. The PATH is ready to go, thanks to the gvm-init.sh file that is sourced by GVM.

On Windows, just crack open a normal Command Prompt window and do:

C:\Users\Test>mklink /j .gvm\groovy\current .gvm\groovy\2.2.2
Junction created for .gvm\groovy\current > .gvm\groovy\2.2.2

C:\Users\Test>mklink /j .gvm\grails\current .gvm\grails\2.3.7
Junction created for .gvm\grails\current > .gvm\grails\2.3.7

Creating a Directory Junction is functionally equivalent to creating a symbolic link and easier to use, because Windows 7 Home Premium constantly complains if I try to do the latter (fixing this would require messing with the security policy, which is a pain in the ass for something that should Just Work):

C:\Users\Test>mklink /d .gvm\grails\current .gvm\grails\2.3.7
You do not have sufficient privilege to perform this operation.

Unfortunately, the gvm current command will not work, but groovy and grails will work fine.

$ gvm c
sh.exe": readlink: command not found
sh.exe": readlink: command not found
sh.exe": readlink: command not found
sh.exe": readlink: command not found
sh.exe": readlink: command not found
sh.exe": readlink: command not found
sh.exe": readlink: command not found
sh.exe": readlink: command not found
sh.exe": readlink: command not found
No candidates are in use

Speedy Disk Imaging

One way to stress test the hell out of a computer, by using an Ubuntu Live USB stick to create a disk image:

  1. Install pigz with sudo add-apt-repository "deb http://archive.ubuntu.com/ubuntu $(lsb_release -sc) main universe restricted multiverse" && sudo apt-get update && sudo apt-get install pigz
  2. Create and compress a disk image using all processor cores at once with dd if=/dev/sda bs=1M | pigz -9cv > disk-image.gz

Watch as all cores saturate with work!

cpu-load

So 6 cores can compress at ~95MB/second:

<stdin> to <stdout> 57241+1 records in
57241+1 records out
60022480896 bytes (60 GB) copied, 634.959 s, 94.5 MB/s

Resulting in a 6:1 compression ratio:

-rw------- 1 ubuntu ubuntu 9308731557 Mar 16 19:12 disk-image.gz