Fei Hu's Blog

Get Better Thinking By Writing

Connect Spark with Cloudera Yarn Cluster

Cloudera automatically installs Spark cluster, but it is not easy to update Spark version by Cloudera. There is another solution for users to run the latest version of Spark on Cloudera Yarn cluste...

Posted by on April 14, 2018

Zeppelin Connects Hive

Make sure that Hive can be accessed remotely using HiveServer2 bash-4.2$ beeline beeline> !connect jdbc:hive2://svr-A3-A-U2:10000 hive hive if unable to access, check t...

Posted by on January 3, 2018

Set up PySpark with Keras

pip install -U matplotlib numpy pandas scipy jupyter ipython scikit-learn scikit-image openslide-python pip install tensorflow pip install git+https://github.com/fcholle...

Posted by on September 11, 2017

Git Notes

Merge a remote pull to the current branch Add the following script to ~/.gitconfig [alias] pr = "!f() { git fetch ${2:-upstream} pull/$1/head:pr/$1 && git checkout pr/$1; }; f" ...

Posted by on July 26, 2017

Parfor in SystemML

org.apache.sysml.api.mlcontext.MLContext -> org.apache.sysml.api.mlcontext.ScriptExecutor -> org.apache.sysml.api.ScriptExecutorUtils -> org.apache.sysml.runtime.controlprogram.Program -&g...

Posted by on July 19, 2017

Install Load Balance V2 for OpenStack

On the controller node apt-get install python-neutron-lbaas -y vi /etc/neutron/neutron.conf: [DEFAULT] ... service_plugins = router, neutron_lbaas.services.loadbalancer.plugin.LoadBalan...

Posted by on July 19, 2017

SystemML Workflow

1. Create DML script Use org.apache.sysml.api.mlcontex.Script class to create DML script. The in() and out() functions are utilized to map the input and output values 2. Handling DML /org/apac...

Posted by on July 10, 2017

CS231n_Notes_1.2: Linear Classification

Interpreting a linear classifier Notice that a linear classifier computes the score of a class as a weighted sum of all of its pixel values across all 3 of its color channels. Depending on precisel...

Posted by on July 8, 2017

CS231n_Notes_1.1: Image Classification

Image Classification Problem is the task of assigning an input image one label from a fixed set of categories. Many other seemingly distinct Computer Vision tasks (such as ojbject detection...

Posted by on July 8, 2017

Setup PyCharm For SystemML with PySpark

1. Install PySpark 1.1 Download Spark package 1.2 Set up SPARK_HOME at ~/.bash_profile export SPARK_HOME="/Users/mac/Documents/Software/spark-2.1.1-bin-hadoop2.7" export PATH="$SPARK_HOME/bin:...

Posted by on June 21, 2017