Saturday 27 February 2016

Network operation team 
Network operation automation
Tools/machines to make the right decision before taking the remediation actions
Network health service is the tool to provide consultant service for tools or human to make the correct decision. 

Knowledge:
Amazon’s network architecture
Amazon's network configuration and management

How to monitor web service?
Levels:
Matrices:
Dashboard GUI Design:

CloudWatch Dashboards

Dashboard Software - Tableau.com‎

Google Analytics

Wednesday 24 February 2016

Amazon Leadership Principles

1. Customer Obsession

Leaders start with the customer and work backwards. They work vigorously to earn and keep customer trust. Although leaders pay attention to competitors, they obsess over customers.

Situation - 2014-Sep, our team started one project with Transport Department, NSW. The project's scope is to investigate the benefits of smart motorway signal control in Sydney. 

Task - My role is the principle project investigator. My task is to deliverer a Transport Model to the client in 3 months. more importantly, This is our first project with the client, my task is also to earn the client's trust. 

Activity - For this project, I found that two most important clients in the project: one is a senior transport modeler Christopher and the other is a general transport manager Martin. They both have strong technical background. So, during these three months, I follow two rules in the communication with clients:
1: Heavy client involvement.
2: Focusing on Data.

Result - I would like to mention some results of our team's work:
1: Clients asked lots of technical questions and involve in some technical decisions.
2: This project was later demonstrated on television to the Premier Mike Baird, as a precursor to the Sydney M4 Smart Motorway Project. 
3: Our team signed a much bigger (3+M) project with the client in the end of 2015.
4: When Christopher changed his job to TransUrban, this client relationship brings us new opptunities.

2. Ownership
Leaders are owners. They think long term and don’t sacrifice long-term value for short-term results. They act on behalf of the entire company, beyond just their own team. They never say “that’s not my job".

Situation - 2015-Oct, our team just finished one research Project "Sydney City-scale Transport Insights (SCTI)" and at that time there was one commercial conference hosted by TransUrban in Sydney. It became a good opportunity for us to introduce this project to the potential client.

Task - However, our biz college and my manager have to join another conference in France. So, I volunteered myself to join the event and presented our project to the Technical Manager TransUrban, even through it is not part of my job and definitely not my strength.

Result - the Technical Manager Micheal got interested in our project and later on introduced her college Peterson (the Transport Manager) to me. Then, we scheduled a 2-Hour meeting and agreed to signed the NDA and start a collaboration project. We are currently in the legal process.  

3. Invent and Simplify
Leaders expect and require innovation and invention from their teams and always find ways to simplify. They are externally aware, look for new ideas from everywhere, and are not limited by “not invented here". As we do new things, we accept that we may be misunderstood for long periods of time.

Situation - In the area of Transport Modelling, we have developed our own innovation in 2015 and published the innovation in the best Transportation Journal. We were awarded the ITS Research 2015 because of the innovation. This work is one of our core competitiveness and I am proud of this work. However, recently I read another trend of innovation. And I think in the long round, the other method is better than ours.

Task - As the transport research leader, I need to decide whether we need to reinvent our method. 

Result - I talked with my manager and we decided to start a small project to reinvent our method. I think it is a correct decision.

4. Are Right, A Lot

Leaders are right a lot. They have strong business judgment and good instincts. They seek diverse perspectives and work to disconfirm their beliefs.

Situation - In Feb 2016, our team decided to step into a new research area named Dynamic Traffic Modelling.

Task - It was my responsibility to set up the technical framework and point out our research opportunity.

Action - A technical framework consists of a number of decisions. If the decision is about Transportation, Algorithms and System Design, I will make decisions and explain to our team. If the decision is about Machine Learning and Mathematics, I will discuss the decision with my senior college. 

5. Hire and Develop The Best

Leaders raise the performance bar with every hire and promotion. They recognize exceptional talent, and willingly move them throughout the organization. Leaders develop leaders and take seriously their role in coaching others.  We work on behalf of our people to invent mechanisms for development like Career Choice.

Not much experience.

6. Insist on the Highest Standards

Leaders have relentlessly high standards - many people may think these standards are unreasonably high. Leaders are continually raising the bar and driving their teams to deliver high quality products, services and processes. Leaders ensure that defects do not get sent down the line and that problems are fixed so they stay fixed.

Situation - In 2012, when I was in my PhD. One of my friend from MIT and I were working on Traffic Model Calibration. The measurement of our job is the error between the model's output and the real-world measurement. Usually, when the error is <20%, we would think the model is valid.

Action - However, both my friend and me were kind of obsessed with reducing the error. We will take about methods and spend days doing experiments to reduce errors even by 0.1%.

7. Think Big

Thinking small is a self-fulfilling prophecy. Leaders create and communicate a bold direction that inspires results. They think differently and look around corners for ways to serve customers.

No experience here.

8. Bias for Action

Speed matters in business. Many decisions and actions are reversible and do not need extensive study. We value calculated risk taking. 

9. Frugality
Accomplish more with less. Constraints breed resourcefulness, self-sufficiency and invention. There are no extra points for growing headcount, budget size or fixed expense.

10. Learn and Be Curious

Leaders are never done learning and always seek to improve themselves. They are curious about new possibilities and act to explore them.

Situation - My career aim is to keep track of new ideas and convert my ideas into products. Following this way, I will think of where I can improve myself in Algorithm, System Design and Data Analytics.

Action: For example, recently I found an online web seminar: BiiTiger.com, the host has been working on Tie-1 Technology company for 10+ years and will explain the design of one hot technique or product every week. If I am not busy, I will spent the whole weekend in following the online course. I plan to follow it in the next 1 year. It will definitely improve my skills in System Design.

11. Earn Trust

Leaders listen attentively, speak candidly, and treat others respectfully. They are vocally self-critical, even when doing so is awkward or embarrassing.  Leaders do not believe their or their team’s body odor smells of perfume.  They benchmark themselves and their teams against the best.

12. Dive Deep

Leaders operate at all levels, stay connected to the details, audit frequently, and are skeptical when metrics and anecdote differ. No task is beneath them.

13. Have Backbone; Disagree and Commit

Leaders are obligated to respectfully challenge decisions when they disagree, even when doing so is uncomfortable or exhausting. Leaders have conviction and are tenacious. They do not compromise for the sake of social cohesion. Once a decision is determined, they commit wholly.

14. Deliver Results

Leaders focus on the key inputs for their business and deliver them with the right quality and in a timely fashion. Despite setbacks, they rise to the occasion and never settle.

Situation - 2014-Sep, our team started one project with Transport Department, NSW. 

Task - My role is the principle project investigator. My task is to deliverer a Transport Model to the client in 3 months. more importantly, This is our first project with the client, my task is also to earn the client's trust. However, I found that I do not have sufficient time to evaluate the model according to the clients' requirement. 

Activity - The good thing is that during the project, our team has shown our professions to our clients. Our relationship is smooth. So, I scheduled a meeting with two senior guys from the client company and proposed to evaluate the model using one method instead of two methods. 

Result - I would like to mention some results of our team's work:
1: Clients agreed with my proposal but asked to explain the evaluation process in the report. It was very reasonable.
2: This project was delivered in time and later demonstrated on television to the Premier Mike Baird, as a precursor to the Sydney M4 Smart Motorway Project. 
3: Our team signed a much bigger (3+M) project with the client in the end of 2015.


Monday 22 February 2016

System Design Blogs

1. 推荐系统的理论
https://www.ibm.com/developerworks/cn/web/1103_zhaoct_recommstudy1/

Keywords:
Demographic-based Recommendation
Cold Start
Content-based Recommendation【新品推荐】
Collaborative Filtering-based Recommendation
User-based Recommendation【主流推荐】
K-Neighbor
Item-based Recommendation
Model-based Recommendation
Weighted Hybridization、Switching Hybridization,Mixed Hybridization

补充:

什么是协同过滤

协同过滤是利用集体智慧的一个典型方法。要理解什么是协同过滤 (Collaborative Filtering, 简称 CF),首先想一个简单的问题,如果你现在想看个电影,但你不知道具体看哪部,你会怎么做?大部分的人会问问周围的朋友,看看最近有什么好看的电影推荐,而我们一般更倾向于从口味比较类似的朋友那里得到推荐。这就是协同过滤的核心思想。
协同过滤一般是在海量的用户中发掘出一小部分和你品位比较类似的,在协同过滤中,这些用户成为邻居,然后根据他们喜欢的其他东西组织成一个排序的目录作为推荐给你。
补充2:
对于 User CF,推荐的原则是假设用户会喜欢那些和他有相同喜好的用户喜欢的东西,但如果一个用户没有相同喜好的朋友,那 User CF 的算法的效果就会很差,所以一个用户对的 CF 算法的适应度是和他有多少共同喜好用户成正比的。
Item CF 算法也有一个基本假设,就是用户会喜欢和他以前喜欢的东西相似的东西,那么我们可以计算一个用户喜欢的物品的自相似度。一个用户喜欢物品的自相似度大,就说明他喜欢的东西都是比较相似的,也就是说他比较符合 Item CF 方法的基本假设

2. 密码保存算法
http://www.zhihu.com/question/20479856
Salt + Hash + Delay

3. 一致性解释
http://blog.csdn.net/duxingxia356/article/details/43992015

4. What really happens when you navigate to a URL
http://igoro.com/archive/what-really-happens-when-you-navigate-to-a-url/

5. REGULAR EXPRESSION
https://regex101.com/

6. 并发处理
Java 基本:http://langgufu.iteye.com/blog/2152608
Java Conditional Lock: http://www.cnblogs.com/skywang12345/p/3496716.html
Java Semaphore: http://www.cnblogs.com/skywang12345/p/3534050.html
Lock-Free Implementation: http://ifeve.com/lmax/


Wednesday 3 February 2016

Combination and Permutation

Concept:

The fundamental difference is:
  • Combination does not care ORDER
  • Permutation cares ORDER
Combination:

1. Without duplicate numbers (Any Length)
*There are 2^n solutions.
  for (int i = 0; i < 2^n; i++) {
    find the solution for i.
  }

 2.  Without duplicate numbers (Only Length k)
        if (solution.size() == k) {
            rst.add(new ArrayList(solution));
            return;
        }        

        for (int i = start; i <= n; i++) {// <=n because we want 1 ~ n in this problem
            solution.add(i);
            helper(rst, solution, n, k, i + 1);
            solution.remove(solution.size() - 1); //Back-track
        }

3. With duplicate numbers (but not allowing replication [each number can be used at most 1 time])
1) Build A HashSet
2) Sort the numbers
    Then, the idea is to avoid the same number in the same level.
    int prev = -1;//Repeat detection
        for (int i = start; i < num.length; i++) {
            if (prev != -1 && prev == num[i]) {
                continue;
            }

            list.add(num[i]);
            sum += num[i];
            helper(rst, list, num, target, sum, i + 1);

            //Back track:
            sum -= num[i];
            list.remove(list.size() - 1);

            //Repeat Detection
            prev = num[i];
        }

4. With duplicate numbers (allowing replication)
1) Build A HashSet
2) Sort the numbers
    Then, the idea is to avoid the same number in the same level.
    int prev = -1;//Repeat detection
        for (int i = start; i < num.length; i++) {
            if (prev != -1 && prev == num[i]) {
                continue;
            }

            list.add(num[i]);
            sum += num[i];
            helper(rst, list, num, target, sum, i);   //allow replication

            //Back track:
            sum -= num[i];
            list.remove(list.size() - 1);

            //Repeat Detection
            prev = num[i];
        }

Permutation:

1. Basic Version (No duplication)
for (int i = 0; i < nums.size(); i++) {
            if (accessed[i] == true) {
                continue;
            }
            
            accessed[i] = true;
            list.add(nums.get(i));
            permuteRecursively(results, list, accessed, nums);
            accessed[i] = false;
            list.remove(list.size() - 1);
        }
2. Basic Version (No duplication) But No recursion
Queue<ArrayList<Integer>> queue = new LinkedList<ArrayList<Integer>>();
        ArrayList<Integer> list;
        for (int num : nums) {
            list = new ArrayList<Integer>();
            list.add(num);
            queue.offer(new ArrayList<Integer>(list));
        }

        while (!queue.isEmpty()) {
            list = queue.poll();
            if (list.size() == nums.size()) {
                rst.add(new ArrayList<Integer>(list));
                continue;
            }
            for (int i = 0; i < nums.size(); i++) {
                if (!list.contains(nums.get(i))) {
                    list.add(nums.get(i));
                    queue.offer(new ArrayList<Integer>(list));
                    list.remove(list.size() - 1);
                }
            }
        }

3. (With duplication)
    Sort nums 
    It is the magic part:

if (list.size() == nums.size()) {
            rst.add(new ArrayList<Integer>(list));
            return;
        }

        for (int i = 0; i < nums.size(); i++) {
            if (mark[i] || (i != 0 && mark[i - 1] && nums.get(i) == nums.get(i - 1))) {
                continue;
            }           
            list.add(nums.get(i));
            mark[i] = true;
            dfs(nums, list, rst, mark);
            list.remove(list.size() - 1);
            mark[i] = false;
        }