Blog do projektu Open Source JavaHotel

piątek, 5 stycznia 2018

Civilization The Board Game, next version

I deployed a new version of my computer implementation of  Civilization The Board Game. The implementation consists of three parts:
New features
  • Greeting window more verbose
  • Research, building up technology tree
Greeting window

Civilizations tiles more descriptive. But that's description only, civilization capabilities, government and technologies are not implemented yet.
Technology tree
The player can spend trade and enhance technology tree.
The current technology tree can be reviewed any time by clicking technology icon. The player can also peek opponent's achievements on this field.
Unfortunately, the thumb is down. The technology abilities are not implemented.
Next steps
Producing unlocked buildings.

środa, 27 grudnia 2017

Next version of Civilization The Board Game

I deployed a new version of my computer implementation of  Civilization The Board Game. The implementation consists of three parts:
New features
  • Player can explore huts
  • Player can attack and take over villages
  • Fight between players in two player game
  • Improvements
Exploring huts
To explore hut, go to the square adjacent to the hut and click "Explore" button.

Then hut is visible in the player resource panel on the left. Click the resource panel and discover the resource guarded by the hut.

Attacking villages
In order to take over the village, the player has to position the figure close to the village and click "Attack" button.

Next step is to start the battle.
Next step, last but not least, is to win the battle!
In training, single player game, the player has to take a role of attacker and defender. In two player game, the opponent has to fight as a barbarian.
To fight, just drag a unit from waiting list to the battle zone. The game engine will resolve the result of the battle.
This time you win. But be careful next time, you may lose as well.
Two player battle
In two player game, one of the players can attack another player. 

The only difference comparing to village battle, the victor can pick up a loot from the loser.
I removed confirmation for all movements ("Are you sure ?"). The game runs more smoothly but there is a risk of mistake. 
The game is running slowly and awkward on Heroku in two players mode. In two players game, the client is probing constantly in 0.5 sec intervals the server to discover the change caused by other player action. The server is under the shell and is reacting slowly causing latency. Some improvements are necessary.
Next steps
  • Research implementation
  • Improvements

czwartek, 30 listopada 2017

HDP 2.6.3, Ambari, Python

I was stuck by very nasty error while installing latest HDP 2.6.3 on RedHat 7.3. Installation failed giving the following error message:
Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/custom_actions/scripts/", line 170, in actionexecute
    installed_packages, repos = self.execute_existing_repos_and_installed_packages_check(config)
  File "/var/lib/ambari-agent/cache/custom_actions/scripts/", line 233, in execute_existing_repos_and_installed_packages_check
    installedPackages = self.pkg_provider.all_installed_packages()
  File "/usr/lib/python2.6/site-packages/resource_management/core/providers/package/", line 222, in all_installed_packages
    return self._get_installed_packages(None)
  File "/usr/lib/python2.6/site-packages/resource_management/core/providers/package/", line 157, in _get_installed_packages
    packages = self._lookup_packages([AMBARI_SUDO_BINARY, "yum", "list", "installed"], "Installed Packages")
  File "/usr/lib/python2.6/site-packages/resource_management/core/providers/package/", line 191, in _lookup_packages
    if items[i + 2].find('@') == 0:
IndexError: list index out of range
No handlers could be found for logger "ambari_agent.HostCheckReportFileHandler"
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/ambari_agent/", line 74, in writeHostChecksCustomActionsFile
TypeError: string indices must be integers, not str
I took me some time to get to the bottom. It looked that the culprit was additional stuff attached to the end of "yum list installed".
yum-metadata-parser.x86_64        1.1.4-10.el7               @anaconda/7.3
yum-rhn-plugin.noarch             2.0.1-6.1.el7_3            @rhel-7-server-rpms
zip.x86_64                        3.0-11.el7                 @anaconda/7.3
zlib.x86_64                       1.2.7-17.el7               @anaconda/7.3
Uploading Enabled Repositories Report
Loaded plugins: product-id
Judging from the Python source code:
    for i in range(0, len(items), 3):

        if '.' in items[i]:
          items[i] = items[i][:items[i].rindex('.')]
        if items[i + 2].find('@') == 0:
          items[i + 2] = items[i + 2][1:]
        packages.append(items[i:i + 3])

    return packages
the installer is expecting a number of elements returned from "yum list" to be divided by 3 and this two lines at the end obviously break this assumption. So the solution (working) was the manual fix in /usr/lib/ambari-agent/lib/resource_management/core/providers/package/ Important: in this file, not in the file specified in error stack trace:  /usr/lib/python2.6/site-packages/resource_management/core/providers/package/, because the installer every time overwrites script in /usr/lib/python2.6/site-packages/resource_management/core/providers/package directory. After the fix the aforementioned piece of code looks:
     for i in range(0, len(items), 3):

        if i + 2 >= len(items) : break

        if '.' in items[i]:
          items[i] = items[i][:items[i].rindex('.')]
        if items[i + 2].find('@') == 0:
          items[i + 2] = items[i + 2][1:]
        packages.append(items[i:i + 3])

    return packages

sobota, 28 października 2017

Next version of Civilization The Board Game

I deployed next version of my computer implementation of  Civilization The Board Game. The implementation consists of three parts:
New features
  • spend trade for production and undo spend
  • scout can send production to the city and undo sending
  • buying units
Spend trade for production

Just specify the number of production you want to get and the trade is reduced accordingly. So you can do shopping as you wish.The player can also undo the last spending unless it is used.
Send production from square to the city

Click the city and choose the scout you want to send production.
The player can also undo this action unless production is spent.
Click Undo button right to the grayed "Send Production" button.
Buying units

The player can buy units although the battle is not implemented yet. The unit panel shows only the number of units of a particular type. After clicking the panel, the detailed list of units including their strength is revealed. Of course, this option is not available for opposite player. Also, in the market panel, any player can see the detailed list of killed units but not the units still waiting to be taken.
Next steps
  • Resource harvesting including friendly villages
  • Battle

czwartek, 26 października 2017

HDP, BigInsights, Kafka, Kerberos

I spent several hours resolving a nasty problem which came up after enabling Kerberos security. Suddenly command line kafka-topic utility tools refused to cooperate:
[2017-10-26 23:31:17,424] WARN Could not login: the client is being asked for a password, but the Zookeeper client code does not currently support obtaining a password from the user. Make sure that the client is configured to use a ticket cache (using the JAAS configuration setting 'useTicketCache=true)' and restart the client. If you still get this message after that, the TGT in the ticket cache has expired and must be manually refreshed. To do so, first determine if you are using a password or a keytab. If the former, run kinit in a Unix shell in the environment of the user who is running this Zookeeper client using the command 'kinit ' (where  is the name of the client's Kerberos principal). If the latter, do 'kinit -k -t  ' (where  is the name of the Kerberos principal, and  is the location of the keytab file). After manually refreshing your cache, restart this client. If you continue to see this message after manually refreshing your cache, ensure that your KDC host's clock is in sync with this host's clock. (org.apache.zookeeper.client.ZooKeeperSaslClient)
[2017-10-26 23:31:17,426] WARN SASL configuration failed: No password provided Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it. (org.apache.zookeeper.ClientCnxn)
Exception in thread "main" org.I0Itec.zkclient.exception.ZkAuthFailedException: Authentication failure
 at org.I0Itec.zkclient.ZkClient.waitForKeeperState(
 at org.I0Itec.zkclient.ZkClient.waitUntilConnected(
 at org.I0Itec.zkclient.ZkClient.connect(
 at org.I0Itec.zkclient.ZkClient.(
 at org.I0Itec.zkclient.ZkClient.(
 at kafka.utils.ZkUtils$.createZkClientAndConnection(ZkUtils.scala:76)
 at kafka.utils.ZkUtils$.apply(ZkUtils.scala:58)
 at kafka.admin.TopicCommand$.main(TopicCommand.scala:53)
 at kafka.admin.TopicCommand.main(TopicCommand.scala)

The reason is quite simple. Kafka to communicate with underlying Zookeper uses /etc/security/keytabs/kafka.service.keytab. As a default, this file has permission 400 so only kafka user can access it.
The solution is to change permission to 440, so the security is softened a little bit but the file is still protected. User vying to create Kafka topic should belong to hadoop group.

poniedziałek, 9 października 2017

Hadoop and SQL engines

Every Hadoop distribution comes with several SQL engines. So I decided to create a simple test to compare them. Run the same queries against the same data set. So far I have been working with two Hadoop distribution, BigInsights 4.x and HDP 2.6.2 with Big SQL 5.0.1. The second is now the successor of BigInsights.
I was comparing the following SQL engines:

  • MySQL, embedded
  • Hive against data in different format: text files, Parquet and OCR
  • Big SQL on Hive tables, Parquet and OCR
  • Spark SQL
  • Phoenix, SQL engine for HBase.
It is not any kind of benchmarking, the purpose is not to prove the superiority of one SQL engine over another. I also haven't done any kind of tunning or reconfiguration to speed up. Just to conduct a simple check after installation and have several numbers at hand.
The test description and several results are here.
Although I do not claim any ultimate authority here, I can provide several conclusions.
  • Big SQL is a winner. Particularly comparing to Hive. Very important: Big SQL is running on the same physical data, the only difference is a different computational model. It even beats MySQL. But, of course, MySQL will get the upper hand for OLTP requests. 
  • Hive behaves much better paired with TEZ. On the other hand, the execution time is very fluid, can change from one execution to another drastically.
  • Spark SQL is outside competition but it is hard to outmatch in-memory execution.
  • Phoenix SQL is at the end of the race, but the execution time is very stable.

sobota, 30 września 2017

Visualize car data with Brunel and Scala

There is a sample in IBM Data Science Experience "Visualize car data with Brunel". But this sample notebook is written in Python, PySpark. So I transformed it to Scala just receiving the same result but using Scala syntax. I added some comments to explain the code.
The result is published here.
To run it:

  • Download Cars+.ipynb notebook
  • Upload to Jupyter with Apache Toree-Scala (Spark) kernel enabled
  • Enjoy