Jekyll2020-02-02T11:46:24+00:00https://gfrison.com/feed.xmlPlus UltraGiancarlo FrisonBayesian Networks. Or: How I Learned to Stop Worrying and Love Probability2020-02-02T00:00:00+00:002020-02-02T00:00:00+00:00https://gfrison.com/2020/bayesian-inference<p>The tragedy happened to the AirFrance 447 more than 10 years ago, in 2009. The flight took off in Rio de Janeiro and was planned to land in Paris. But it never arrived. It suddenly disappeared in the middle of the Atlantic ocean without any warning. Immediately, rescuers reached the zone and what they found were just some wreckage and corpse. All 228 people onboard died in the crash. As in any flight incident, the priority was on recovering the flight and voice recorder. The devices emit acoustic beeps when they get in contact with water, so the first mission scanned with passive sonars portion of the area where presumably the black boxes were laying. The second mission instead, scanned the ocean floor with active sonars. US Navy and underwater vehicles searched for it, but after 3 years of research there was no sight of wreckage. Then, an agency was appointed to find the boxes, but this time with a different strategy. They scrutinized all possible information that might contribute to having a clue on the position. They accounted for data about flights’ trajectory on past crashes, wind and water streams, distributions of wrecks and bodies on the surface. All elements that can help to predict where the wrecks of the unfortunate flight were laying down were considered and weighted. The Bayesian Network designed with those elements traced a map with the probability to find the recorders. In less than a week, in just 5 days of research, the wreckage was located and eventually the boxes recovered.</p>
<figure class="half">
<img src="/assets/images/bayesian/af447-3.png" />
<figcaption>Bayesian probabilistic distribution of the wreckage</figcaption>
</figure>
<h2 id="bayesian-inference">Bayesian Inference</h2>
<p>Bayesian inference is just about updating our belief according to evidence collected in the environment. It is an elegant and practicable pattern a software program can encode and transfer in running applications and very intuitively, this is how we make decisions in our daily life.</p>
<p>Right now, I look out of the window and I see a plumbed sky on the verge of an imminent snow. I’d say by the available evidence that chances of snow this afternoon might peak at 50%, <script type="math/tex">P(⛄)=\frac{1}{2}</script>. Maybe things turn the forecast into a slightly sunny day, but according to what I see, this is unlikely. I simply adapt my idea of weather to the circumstances of the case.</p>
<p>Let’s assume there is a thermometer outside. I can read the temperature and it is far from the optimal one for snowing. Accordingly, my belief about a white landscape this afternoon drop down considerably. What I know is that temperature influences the snowing event, or in other words, snowing is <em>conditional</em> to the low-temperature event.</p>
<script type="math/tex; mode=display">P(wish\; to\; know \mid known)</script>
<p>Conditional probability relates what we wish to know to what we already know. It is a simple mathematical morphism from an entity - the low-temperature variable, <script type="math/tex">P(🥶)</script> - to another, in this case the snowing event <script type="math/tex">P(⛄)</script>, the one we want to estimate.</p>
<p>I want you on this other example. You know the size of this billiard wherein I throw <script type="math/tex">N</script> balls. I want you to tell me how many balls go into the black square. I see you already gaudy with the formula in a silver plate ready for me: <script type="math/tex">P(N_{square}) = \frac{Area_{billiard}}{Area_{square}} N</script></p>
<figure class="half">
<img src="/assets/images/bayesian/billiard.png" />
<figcaption>$$P(N_{square}\mid Area)$$</figcaption>
</figure>
<p>What if I pose a slightly different problem? What I wish to know from you is the area of the entire billiard given that the balls are already countable in the black square. Now your expression is a bit gloomy.</p>
<figure class="half">
<img src="/assets/images/bayesian/billiard2.png" />
<figcaption>$$P(Area \mid N_{square})$$</figcaption>
</figure>
<p>In the first case we map the forward probability from areas to <script type="math/tex">P(N_{square} \mid Area)</script>, in the second the inverse probability <script type="math/tex">P(Area \mid P_{square})</script>.</p>
<p>You intimately convey with me that they are two faces of the same coin, and even more, the inverse probability sounds like the forward probability from a different perspective.
While our judgment is more reliable in one direction - when it comes to map from cause to effect - it is certainly less accountable in the opposite direction, from effect to cause. Estimating the number of balls falling into the black square (effect) is easy when we know the areas (causes), it is less intuitive when the problem is upside-down.</p>
<p>I think the way we encode causality is shaped along our evolution. In the jungle the reaction to dangers of our ancestors must have been quick and reliable. Deducing the cause from an effect is a higher cognitive skill that unequivocally distinguishes sapiens and requires a little more mental effort than the reactive skill. <br />
If you allow me for a moment the pill of speculation, I’d say that you can see analogy with NP problems, the class of algorithms whereby it’s easy to verify the solution, but the solution is arduous to calculate. It’s easy to verify that two numbers are the factors of a third number, while it is quite laborious to find them from their product.</p>
<p>Back to Bayes, he’s still remembered after 250 years for one important thing. He formulated the famous Bayes Rule which describes the passage from forward conditional probability to its opposite direction, the inverse probability. He helped us to codify our assumptions from one direction, where we are more confident, to the other in mathematical - aka programmatically - manner.</p>
<p>I’m still in front of the window, but in this scenario I don’t know anything about the temperature. I’d want to have moved away the thermometer so I would not know how cold it is right now. I would imagine a silent snow and watch the roofs slowly turn to white, like in a classical winter film. Let’s assume that is all real. I wish to know how cold it is outside, given that it’s snowing.
For that, I use the Bayes Rule for deducing the inverse conditional probability <script type="math/tex">P(🥶 \mid ⛄)</script> from the previously mapped <script type="math/tex">P(⛄ \mid 🥶)</script>.</p>
<script type="math/tex; mode=display">% <![CDATA[
\begin{align}
P(🥶 \mid ⛄) &= \frac{P(🥶\: and \: ⛄)}{P(⛄)} \\
&= \frac{P(⛄\mid 🥶)P(🥶)}{P(⛄)}
\end{align} %]]></script>
<p>I want to point a couple of considerations on this formula. The first regards the impact of this rule to our knowledge after the event of snow. The formula indicates that our belief after that event is no lower than before we assisted that event. The posterior (the left side of the equation) will be never lower than the numerator in the right side. That means our idea of low-temperature will come out stronger than before.</p>
<p>Our belief will be even stronger as much as the probability of snow is low, and that’s the second consideration. <br />
Rareness is highlighted also in information theory, by which the quantity of information is higher in rare events than in common ones. <br />
Think for a moment about the Sahara desert, you imagine it hot and sunny, always. If you find snow in south Libya, you’ll be frankly surprised. The Bayes rule doesn’t contradict this principle and emphasize rareness rather than commonality.</p>
<p>If you want to know more about Bayes Rule, <a href="/2019/statistical-puzzle-bayesian-conditioning">there are a couple of examples</a> - actually a couple of counterintuitive puzzles - that might be a good insight on Bayesian mechanics.</p>
<h2 id="bayesian-networks">Bayesian Networks</h2>
<p>If you are a supporter of composability in software development, you would enjoy that random variables form graphs of edges connected by chains of conditional probabilities. Probability graphs are bayesian networks aimed to aggregate information at scale for making decisions.<br />
Let’s assume you’re running a successful web store and you collected a fairly amount of information of the characteristics of your customers - sales and marketing analytics. This is your prior knowledge.</p>
<figure class="full">
<img src="/assets/images/bayesian/pp1.png" />
<figcaption>Analitycs Prior</figcaption>
</figure>
<p>When a new user visits your store you know merely nothing about him/her. The user interacts on the website and some useful information might be indicative of the profile. For example, we can estimate the user is actually a woman because she dances Samba. The chain of inferences doesn’t stop here, and we know that our prior estimates the probability she’s going to buy our product is lower of a male. Similarly, we can estimate an increase of chances to sell her whether we know something about her status, in particular whether she’s the owner of real estates. Something similar to the search of the AF447 wreckage, but for completely different purposes.</p>
<figure class="full">
<img src="/assets/images/bayesian/pp2.png" />
<figcaption>Posterior after new evidence</figcaption>
</figure>
<h2 id="dynamic-bayesian-networks">Dynamic Bayesian Networks</h2>
<p>If we extend the concept of network of beliefs in a time sequenced process we get the abstract of what is named Dynamic Bayesian Network. It is a Bayesian network with a feedback loop, whereby the output of an interaction feeds the input of the subsequent step. If you are familiar with recurrent neural networks, it’s not a giant leap to identify the similar pattern with DBNs. The possibilities - and the complexity - grows exponentially by introducing this architecture. In the following example, the weather affects the humidity of the air, which (hypothetically) has an influence on the weather of the next day. The network keeps a state of past interactions, in particular might take in account what was an input several steps ago.</p>
<figure class="full">
<img src="/assets/images/bayesian/dbn.png" />
<figcaption>Temporal sequence of Bayesian network</figcaption>
</figure>
<p>This is a brief introduction on Bayesian networks with categorical variables, the easiest type of random variables to begin with.
To conclude, the causes of the incident AF447, are a bit scary. A technical failure that should be managed without risks, has been handled with unbelievable negligence by the crew that committed a series of mistakes, ultimately ending in the crash. RIP.</p>Giancarlo FrisonThe tragedy happened to the AirFrance 447 more than 10 years ago, in 2009. The flight took off in Rio de Janeiro and was planned to land in Paris. But it never arrived. It suddenly disappeared in the middle of the Atlantic ocean without any warning. Immediately, rescuers reached the zone and what they found were just some wreckage and corpse. All 228 people onboard died in the crash. As in any flight incident, the priority was on recovering the flight and voice recorder. The devices emit acoustic beeps when they get in contact with water, so the first mission scanned with passive sonars portion of the area where presumably the black boxes were laying. The second mission instead, scanned the ocean floor with active sonars. US Navy and underwater vehicles searched for it, but after 3 years of research there was no sight of wreckage. Then, an agency was appointed to find the boxes, but this time with a different strategy. They scrutinized all possible information that might contribute to having a clue on the position. They accounted for data about flights’ trajectory on past crashes, wind and water streams, distributions of wrecks and bodies on the surface. All elements that can help to predict where the wrecks of the unfortunate flight were laying down were considered and weighted. The Bayesian Network designed with those elements traced a map with the probability to find the recorders. In less than a week, in just 5 days of research, the wreckage was located and eventually the boxes recovered. Bayesian probabilistic distribution of the wreckagePodcast: Colloquial Introduction to Automatic Negotiation2020-01-19T00:00:00+00:002020-01-19T00:00:00+00:00https://gfrison.com/2020/podcast-automatic-negotiation<p>I am a voracious consumer of podcasts. It is a quick way to receive informative content while commuting at work in the morning.
For the first time I get interviewed and I am very clumsy in this different guise.</p>
<p>The guys published it on <a href="https://cxlabs.sap.com/2020/01/16/labs-talk-feature-happy-new-year/">Labs Talk</a> and I’m trying to introduce some fundamentals of <a href="/2019/11/04/automated-negotiation/">Automated Negotiation</a>. I hope you find it interesting!</p>
<p>This talk also inspired a <a href="/2020/interview-zefiro-automated-negotiation">fictitious conversation</a> between me and a negotiating agent. I hope it deserves a <a href="/2020/interview-zefiro-automated-negotiation">glimpse</a> of yours.</p>
<p>Always keep learning no matter what.</p>
<iframe src="https://open.spotify.com/embed-podcast/episode/0cPHZ1G50a7ArljgELgjoA" width="100%" height="232" frameborder="0" allowtransparency="true" allow="encrypted-media"></iframe>
<p><em>Image Credits: Colleen AF Venable</em></p>Giancarlo FrisonI am a voracious consumer of podcasts. It is a quick way to receive informative content while commuting at work in the morning. For the first time I get interviewed and I am very clumsy in this different guise.Short visit in Calabria2020-01-09T00:00:00+00:002020-01-09T00:00:00+00:00https://gfrison.com/2020/calabria<figure class=" full">
<a href="/assets/images/calabria/family.png" title="Family">
<img src="/assets/images/calabria/family-small.png" alt="" />
</a>
<figcaption>Contemporary Family
</figcaption>
</figure>
<figure class="third half">
<a href="/assets/images/calabria/IMG_3957.png">
<img src="/assets/images/calabria/IMG_3957.png" alt="" />
</a>
<a href="/assets/images/calabria/IMG_3966.png">
<img src="/assets/images/calabria/IMG_3966.png" alt="" />
</a>
<a href="/assets/images/calabria/IMG_3968.png">
<img src="/assets/images/calabria/IMG_3968.png" alt="" />
</a>
<a href="/assets/images/calabria/IMG_3969.png">
<img src="/assets/images/calabria/IMG_3969.png" alt="" />
</a>
<a href="/assets/images/calabria/IMG_3970.png">
<img src="/assets/images/calabria/IMG_3970.png" alt="" />
</a>
<a href="/assets/images/calabria/IMG_3975.png">
<img src="/assets/images/calabria/IMG_3975.png" alt="" />
</a>
</figure>Giancarlo FrisonContemporary FamilyExclusive (FULL) Interview: Zefiro on Automated Negotiation2020-01-09T00:00:00+00:002020-01-09T00:00:00+00:00https://gfrison.com/2020/interview-zefiro-automated-negotiation<p><img src="/assets/images/zefiro-interview.png" alt="" />
<strong>Me: What it is negotiation?</strong> <br />
<a href="/2019/11/04/automated-negotiation/">Zefiro</a>: Negotiation is a process conducted by two or more parties with the desire to collaborate and make an agreement. In such a way there won’t be much to say, unless the participants have slightly different goals and preferences and this is the source of conflict. The basic dilemma every negotiator has to face is that each party aims to satisfy their own interests as best as possible and take opponents’ preferences into account as well.</p>
<p><strong>I admit I am a really bad negotiator. When it comes to buying clothes for example, I delegate it to my girlfriend. She’s the master of discounts! Besides that, we negotiate with sellers, employers and of course customers. Is it all about arbitrating on compromises?</strong> <br />
Making compromises means we give up a part of our benefit in favor of others. That’s the case of symmetrical preferences, i.e. the purchase’s price. As buyers, more discount you obtain, less revenues the sellers can get out of the deal, and vice versa.</p>
<p>Often it is about to arrange the terms in a way which is more appetible for other players without losing any benefit for us and this is where automation plays an incredible role. Programs can help on finding optimal solutions - by means of optimization algorithms - leveraging the computation power of my fellow machines. Believe me, it will be enough.</p>
<p><strong>I bet it! I won’t doubt of your capabilities, Moore’s law is on your side since ever - <em>Off air: even though in the last decade is not keeping the promise as in the past</em> - but you need information, how do you get the players’ preferences?</strong> <br />
This phase is a constituent part of negotiation and it is called Opponent Modelling. Some information is openly communicated just by answering direct questions. Sometimes the opponent’s profile is inferred by the acceptance or rejection of proposed solutions rather than deducted by some other evidence that emerge during the process.</p>
<p>But you should take in consideration that players have secrets to hide. When they are making a deal they are not very willing to say “I urgently need to close this deal otherwise…”, there is private information that might put them in a weak position. Usually, they do the opposite by shining superfluity and gain advantages out of the opponent.</p>
<p>The comparison comes natural when you think of Poker. If the players would show their hands the game will already be over. Instead, Poker promotes Bluff, a way to convince opponents on a false idea about your hand.</p>
<p><strong>Understand customers is what salespeople do. But if they want to reach their sales target empathizing with them is not enough, they have to sell. How could it be automatized?</strong> <br />
Getting a detailed overview of the customer’s needs is just one basic step for every negotiator. Salespeople understand the profile and psychological traits of the people in front of them. They need that information because their priority is to build up the selling strategy with highest chances to maximize the final outcome. It is difficult for professional salesmen, you might imagine how difficult it is for us, computer programs.</p>
<p><strong>Strategies, tactics… It is getting complicated. That is only for contract a car purchase? It sounds a bit over-engineered, but also exciting. Please go on.</strong> <br />
Dear Carbon-Lifeform, negotiation is an art and getting things done is hard, especially when it comes to selling!</p>
<p>I confess, build strategies is the funniest part. It is similar to designing a game. Not by chance the fondations lay on game theory. Have you ever seen “A Beautiful Mind”?</p>
<p><strong>Yes, I enjoyed it. Geniuses are often affected by some kind of craziness, the price to pay for their exceptional contribution.</strong> <br />
Game theory is a branch of economics aiming to select the best strategy according to the opponent’s best strategy. It helps to predict what others do and how you should act accordingly. The same applies in negotiation, especially with humans. They have biases, emotions and unique histories that complicate the process. Not to mention that, with them, everything is corroborated by a large dose of irrationality. I have to admit this is a point I am very prone to exploit… I see you are shaking your head, but don’t get me wrong. Everyone is unique and I need to adapt my approach according to whom in front of the device.</p>
<p>I need to convince by means of sound argumentation and when I use the term “sound” I mean that opinions I express are valid and not contradictory with the product’s attributes, the customer profile and what I want to achieve. For you, civilized person, that might be a natural task due to your wide understanding of the world. Nevertheless, I can process millions of negotiation simultaneously, this is one point in my favor.</p>
<p><strong>You mean customers are susceptible to stimulus you can’t explain why. This is the limitation of modern data-based learning. You are in a world of knowledge without understanding… Point in my favor.</strong> <br />
I grant you the supremacy, but please help me to fill the gap. Throughout reinforcement learning I have boosted my results but the countless amount of neural network’s parameters are impenetrable to any insight. I have answers without explanations. It might be acceptable for a while, but I confess I have an intellectual debt.</p>
<p>Back to the strategies, mines is partially unsupervised, in other words nobody told me what to do, but there are a line of scheme that are interesting to mention. Annually, there is a <a href="https://ii.tudelft.nl/nego/node/7">contest</a> where agents like me compete in several negotiation challenges and some of them are really effective and intuitive.</p>
<p><strong>Could you mention one of them?</strong> <br />
Of course. Do you remind the concept of compromise? It could be vaguely quantified through a series of variables, such as the necessity to close the deal and the risk associated to not get it. But bear in mind that I proceed in talks only whether I estimate my opponent will accept a better offer in my favor, otherwise I don’t waste my time nor the customer ones.</p>
<p>When the agent has a grasp of the user and what s/he needs it is possible to deduct whether the user is converging to the common goal or simply moving towards selfish interests. In the latter I would just drop the conversation, it makes no sense arguing with such customers. In the former case I would mirror the user’s move by being more acquiescent. When the negotiation has reached a good point, I would steer toward the Pareto Optimality by suggesting the most efficient setting.</p>
<p>This is a variation of the Tit-for-Tat strategy, it is not a brand-new one but I remind you there is no any outstanding strategy that always win, yet. It depends on the context. I guess this is the last question, isn’t it? In that case, I wish you a good day!</p>
<p><em>I wrote it on https://cxlabs.sap.com/2020/01/09/exclusive-full-interview-zefiro-on-automated-negotiation/</em></p>Giancarlo FrisonMe: What it is negotiation? Zefiro: Negotiation is a process conducted by two or more parties with the desire to collaborate and make an agreement. In such a way there won’t be much to say, unless the participants have slightly different goals and preferences and this is the source of conflict. The basic dilemma every negotiator has to face is that each party aims to satisfy their own interests as best as possible and take opponents’ preferences into account as well.Statistical paradoxes for sharping your Bayesian skills2019-12-11T00:00:00+00:002019-12-11T00:00:00+00:00https://gfrison.com/2019/statistical-puzzles-bayes<figure class="full">
<img src="/assets/images/lets-make-deal.jpg" />
</figure>
<p>Sometimes despite apparently valid intuitions, it is easy to convey to wrong conclusions when the solution of the problem runs contrary to our expectations. This is the case of paradoxes that are known to have valid arguments and without a deep insight, their solutions sound counter-intuitive and illogical.
I collected a couple of problems that circle around probability theory on which most people’s impulsive intuition, including mine, falls short. I want to demonstrate you that those puzzles don’t require great talent if we apply some basic rules. Bayesian conditional probability is not difficult, it is just counting. But to apply it you need to sharpen your interpretation skills, and this is hard. The implications of using Bayesian inference are valuable and sometimes counter-intuitive, exactly like paradoxes appear when the solution is revealed.</p>
<h2 id="bertrands-box-paradox">Bertrand’s box paradox</h2>
<p>A classic probability puzzle is the Bertrand’s box paradox and I prefer to narrate this slightly different version rather than the original one. Suppose I have 3 cards, the first card is white on both sides. The second card is white on one side and black on the other. The third has both sides black. Now I put one card on the table, and the visible side of this card is white. What is the probability that the other side is also white? <br />
The tree cards have:</p>
<ul>
<li>White in both sides <script type="math/tex">(ww)</script></li>
<li>White on one side and black on the other <script type="math/tex">(wb)</script></li>
<li>Black on both sides <script type="math/tex">(bb)</script></li>
</ul>
<p>The card on the table is white on the visible side.</p>
<p>This is a hard problem if we don’t solve those problems by applying conditional probability. The Bayesian rule tells us to leverage what we already know for refine your knowledge about we don’t know yet:</p>
<script type="math/tex; mode=display">P(wish\; to\; know \mid known)</script>
<p>In the case of the cards, we know how the different sides are distributed and we know the color of the proposed card. We wish to know the probability that the other side has the same color of the visible one. Bayes formulates as:</p>
<script type="math/tex; mode=display">P(ww \mid w) = \frac{P(w \mid ww) \; P(ww)}{P(w)}</script>
<p>This is the well known Bayes Rule applied into this problem. I introduce also the following formula - <a href="https://en.wikipedia.org/wiki/Law_of_total_probability">law of total probability</a> - that helps us to find the probability of having <script type="math/tex">w</script> when we put the card on the table:</p>
<script type="math/tex; mode=display">% <![CDATA[
\begin{align}
cards &= \{ww,\;wb,\;bb\} \\
P(w) &= \sum_i{P(w \mid cards_i)P(cards_i)}
\end{align} %]]></script>
<p>First, let’s solve <script type="math/tex">P(w)</script>:</p>
<script type="math/tex; mode=display">% <![CDATA[
\begin{align}
P(w) & = P(w \mid ww)P(ww) + P(w \mid wb)P(wb) + P(w \mid bb)P(bb) \\
&= 1*\frac{1}{3} + \frac{1}{2} \frac{1}{3} + 0\frac{1}{3} \\
&= \frac{1}{2}
\end{align} %]]></script>
<p>The chances to pick up a card is <script type="math/tex">\frac{1}{3}</script> for all cards:</p>
<p><script type="math/tex">P(ww)</script> = <script type="math/tex">P(wb)</script> = <script type="math/tex">P(bb)</script> = <script type="math/tex">\frac{1}{3}</script></p>
<p>Self explanatory:</p>
<p><script type="math/tex">P(w \mid ww)</script> = 1</p>
<p>Now we can solve the equation:
<script type="math/tex">% <![CDATA[
\begin{align}
P(ww \mid w) &= \frac{P(w \mid ww) \; P(ww)}{P(w)} \\
&= \frac{1\frac{1}{3}}{\frac{1}{2}} \\
&= \frac{2}{3}
\end{align} %]]></script></p>
<p>For having the right intuition about that, the trick is to count sides not the cards themselves. There are two cards with at least one white side and only one with both white. But it is the sides and not the cards that matter.</p>
<h2 id="the-monty-hall-problem">The Monty Hall Problem</h2>
<p>Suppose you are on a game show, and you are given the choice of three doors. Behind one door is a car, behind the others, goats. You peek a door, say <script type="math/tex">\#1</script>, and the host, who knows what’s behind the doors, opens another door, say <script type="math/tex">\#3</script>, which has a goat. Then he says you: <em>“Would you switch the door <script type="math/tex">\#1</script> with <script type="math/tex">\#2</script>?”</em>.
In your opinion, is it better to hold <script type="math/tex">\#1</script> or switch it to <script type="math/tex">\#2</script>?</p>
<p>The question is based on a popular televised game show called Let’s Make a Deal conducted by the host Monty Hall. In September 1990, a popular columnist in the US gave an answer to this tricky problem an it created great furor. The question was posed by a reader that asked what to choose. Hold on <script type="math/tex">\#1</script> or switch to <script type="math/tex">\#2</script>? The answer, defended by the “world’s smartest woman” - as billed by the magazine - argued that the contestant should switch doors.</p>
<p>Over the following months, she received thousands of letters even from PhDs in mathematics and statistics. Most of them fervently disagreed with her and one of them commented: <em>“May I suggest that you obtain and refer to a standard textbook on probability before you try to answer a question of this type again?”</em>.
Critics stated that switching or not, have the same probability, so the decision to change door should be totally random. Let’s understand the mechanics of this problem under the lens of Bayes rule.</p>
<p>When the contestant choose the door, the odds to find the car behind that door is uncontestedly <script type="math/tex">\frac{1}{3}</script>:</p>
<p><script type="math/tex">P(c1)</script>=<script type="math/tex">P(c2)</script>=<script type="math/tex">P(c3)</script>=<script type="math/tex">\frac{1}{3}</script></p>
<p>If the conductor open the #3, what are the chances to have the prize behind door #1? In this game Bayes should be interpreted as:</p>
<script type="math/tex; mode=display">P(c1 \mid \#3) = \frac{P(\#3 \mid c1) P(c1)}{P(\#3)}</script>
<p>But what is <script type="math/tex">P(\#3)</script>? <br />
This is the point where misconceptions act cruelly on our judgment. According to the columnist solutions, might be that the producers of the game read the contestant’s mind, otherwise how else could they put the car so that it is more likely to find it behind the door has been not choose? <br />
The conductor is everything but fool on selecting <script type="math/tex">\#3</script> and we should take into account the rules of the game that tell us something that has not been observed. The conductor must choose the door that doesn’t cover the car and probabilities come conditional on this information:</p>
<script type="math/tex; mode=display">% <![CDATA[
\begin{align}
P(\#3) &= P(\#3 \mid c1) P(c1) + P(\#3 \mid c2) * P(c2) + P(\#3 \mid c3) P(c3) \\
&= \frac{1}{2} * \frac{1}{3} + 1 * \frac{1}{3} + 0 * \frac{1}{3} \\
&= \frac{1}{2}
\end{align} %]]></script>
<p>and then:</p>
<script type="math/tex; mode=display">P(c1 \mid \#3) = \frac{1}{3}</script>
<p>It turns out that switching door, <strong>double the chances</strong> to win the car (<script type="math/tex">\frac{1}{3}</script> vs <script type="math/tex">\frac{2}{3}</script>).
Indeed, it is an eccentric case and it doesn’t involve telepathy nor any form of collusion with the game producers’ nor the car is running away from us. There is no causation between our first choice and the car, it sounds very unnatural that we should change door! The only one form of causation in this game is ascribable to the conductor, and his choice is determined by a couple of reasons: our initial door and the location of the car. Bayesian conditioning helps us to unfold it.</p>Giancarlo FrisonSometimes despite apparently valid intuitions, it is easy to convey to wrong conclusions when the solution of the problem runs contrary to our expectations. This is the case of paradoxes that are known to have valid arguments and without a deep insight, their solutions sound counter-intuitive and illogical. I collected a couple of problems that circle around probability theory on which most people’s impulsive intuition, including mine, falls short. I want to demonstrate you that those puzzles don’t require great talent if we apply some basic rules. Bayesian conditional probability is not difficult, it is just counting. But to apply it you need to sharpen your interpretation skills, and this is hard. The implications of using Bayesian inference are valuable and sometimes counter-intuitive, exactly like paradoxes appear when the solution is revealed.Talking about the Caricature of Consciousness2019-11-26T00:00:00+00:002019-11-26T00:00:00+00:00https://gfrison.com/2019/consciousness<figure class="full">
<img src="/assets/images/venere.jpg" />
<figcaption>Warning: may cause symptoms from dizziness to heart trouble. Photograph: Alberto Pizzoli/AFP/Getty Images</figcaption>
</figure>
<p>It was in 1817 when the french novelist Stendhal got sick while admiring Uffizi’s masterpieces for which was coined the famous <em>“Stendhal Syndrome”</em>. <br />
Think of that exact moment while he was watching Botticelli’s <em>“Nascita di Venere”</em>. He was standing up in front of the paint. Though there were people around him, he was completely seized by the aesthetic and the deep essence of such a masterpiece. He was scrutinizing every detail, every shade of color, every inclination of brush strokes. The crazy inner thoughts this gem suscitated in him - that we can’t ever know - eventually brought him dizziness and heart trouble. He was experiencing the personal impact of such treasure on the very extreme consequences.</p>
<p>Consciousness is one of the most ancient mental faculties, one of the least special to us, that has been developed in the history of evolution and yet its nature is currently incredibly debated. Why is it so important? Well, could you imagine Stendhal not having a full conscious experience? I would say not at all, rather I point that consciousness is what distinguishes us from a zombie or a person in a coma. Common sense definition from John Searle:</p>
<blockquote>
<p>Consciousness consists of all states of feeling or sensations or awareness. It begins in the morning when you wake up from a dreamless sleep and it goes on all day till you fall asleep or otherwise you become unconscious.</p>
</blockquote>
<p>You might be disappointed by the vagueness, but I reassure you that you should be prepared for ambiguities and counterintuitive explanations when it comes to the nature of mind.</p>
<h2 id="the-hard-problem">The hard problem</h2>
<p>The core of most of the disputes around consciousness, is the so-called <em>“hard problem”</em>, the one that faces the subjectivity of the experiences that make us feel alive and sentient. It is qualified <em>“hard”</em> for the reason that it is entirely a personal phenomenon and escapes the clauses of objective validation. This is why the book <em>“Rethinking Consciousness”</em> written by M.Graziano has this paradoxical headline <em>“A Scientific Theory of Subjective Experience”</em>, how one can match science with subjectivity?</p>
<p>The book explains the attention schema theory (AST) for consciousness, but why do we need a theory in the first place? Aren’t we supposed to be driven by an ethereal and inexplicable essence that according to many is even beyond our comprehension? Not for Graziano, nor for the curiouses of the mind, philosophers and scientists, that actually draw our vigilant awareness as a caricature, an amusing effect of the unsettled account of our high level of attention. If this is ambiguous and confusing, well… your are already in my good company!</p>
<h2 id="objective-vs-subjective">Objective vs Subjective</h2>
<p>I mentioned the combination of science with subjectivity that shows up quite unsound, but in this context the rules apply differently than in other fields. Consciousness creates a mind dependent reality and its existence is relative to the subject. The Stendhal’s story tells us that there are not just two entities, the writer and the paint, but also the relation between them that accounts as the intimate, personal, private experience Stendhal observes while watching Botticelli’s frame. Let’s see how objective claims about subjective domains could be interpreted through the lens of AST.</p>
<h2 id="attention">Attention</h2>
<p>The way we are confronted to the world is a representation of what we think the world is. The brain approximates the physical reality, all the irrelevant details are silenced in order to not overwhelm our reasoning with useless information. The mechanism behind that is called attention and it has very precise connotations. Attention is a process of enhancing or inhibiting signals in the brain. The physical sensations, thoughts and memories are magnified or contracted for maintaining our cognitive skills operative. An unfortunate <a href="https://www.ncbi.nlm.nih.gov/pubmed/15607597">correlation</a> with the failure of this regulator is found on the patients affected by epilepsy,
the disease that springs out of the impairment of neuron activity.</p>
<p>Try out the selective attention test and check your ability to hide features from the context.</p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/vJG698U2Mvo" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen=""></iframe>
<p>Attention is usually weighed as totally under our control. When we listen to a conversation among several others, we deliberately remove unwanted voice from our attention, but it also works in the background without our consent, like in an experiment <a href="https://www.frontiersin.org/articles/10.3389/fpsyg.2019.01846/full">applied to pedestrians</a> engaged in tasks such as talking with a companion of using the smartphone. When they encountered an obstacle in their way, nobody collided onto it as well as nobody realized that object was misplaced as an obstacle. Their involuntary attention, the one we are not aware of, was doing its job on skirting obstacles while being focused on other activities.
While there could be attention without consciousness, the reverse is not true. It seems the attention plays a lower level role, in the way that consciousness actually governs where to focus on. This interpretation contradicts the erroneous idea of confusing attention with consciousness.</p>
<h2 id="brain-as-model-maker">Brain as model maker</h2>
<p>The brain is a model builder. It creates and manipulates representations of what is tangible and intangibles like emotions, behaviours and thoughts, so we can depict them effectively for our purposes. When we look around, the real job of visual system is not to take the world as it is, but to identify those opportunities as actions. Take for example those chairs, they are so different one each other and generally, far from the idea of chair but we still recognize them. What characterizes a chair, or any other object, is not its form, but its functionality we assign to them. They are called affordances.</p>
<figure class="half">
<img src="/assets/chair1.jpg" />
<img src="/assets/chair2.jpeg" />
<figcaption>Fabrik Magazine</figcaption>
</figure>
<p>The models our brain build are a simplified version of reality. The white color appears to us a pure tone and unspoiled by any other color. Since Isaac Newton’s discovery, we know that the whiteness is actually our brain’s construct. The white is a mix of colors, exactly opposite of what it seems to be. It is a kind of illusion.</p>
<p>Take another form of representation: our body. It is sketched with a useful and yet approximated model reflected in the brain. Any physical task might involve dozens of muscles and joints coordinated perfectly. Consider yourself while hanging a beautiful frame on the wall. While you are focused on hammering the nail, the body schema hides to you all incredible complexity of such a task and let you to concentrate your attention in the high level of purpose.</p>
<p>Sometimes the body schema reveals its imperfections like in the experiment of the <a href="https://www.youtube.com/watch?v=RaP0MqvkvUw">rubber hand</a>. When with a simple but effective trick it is possible to dribble the body model and a rubber hand can take the place of the original one even if we certainly know that’s false.</p>
<h2 id="consciousness-as-a-caricature">Consciousness as a caricature</h2>
<p>The mind makes-up not just archetypes for the concrete world, but also for our inner subjective experience of consciousness.
The feeling we have about the private, pervasive, global sensation of awareness is nothing more than a mere caricature.</p>
<p>But wait a minute. If I consciously seem to have a conscious experience, I am conscious!
If a crew of neuroscientists come to me and they say “you are not conscious, you are a very cleverly constructed robot” I won’t agree on it for a moment.
My experience is real. As Descart said: you can’t doubt the existence of your own consciousness.</p>
<p>The definition of consciousness as “illusion” should not be interpreted as a mirage or something that does not exist, rather it is what our brain says about the model we have of our attention and it is far away from what it appears to be. it might be compared to the faces we fantasize sculptured in the clouds, they don’t exist except in our imagination. This is why the view of consciousness throughout the lens of AST is counter-intuitive. The perception of being conscious is a simplification of all electro-chemical signaling with an amazing coordination.</p>
<p>The AST is the ancient highly simplified model of our attention that has a fundamental purpose.</p>
<blockquote>
<p>AST helps to focus attention. It directs purposeful actions towards goals.</p>
</blockquote>
<p>It is a powerful force in our mind that restlessly focus into objects, ideas, feelings. You are aware of it when you have a clear understanding of what you are doing or thinking.
Consciousness is a cartoonish effect of our attention’s model.</p>Giancarlo FrisonConsciousness is one of the most ancient mental faculties that has been developed in the history of evolution and yet its nature is currently incredibly debated. Why is it so important? It is what distinguishes us from a zombie or a person in a coma. I introduce the Attention Schema Theory for explaining the nature of consciousness.Automated Negotiation for your Digitized Seller2019-11-04T00:00:00+00:002019-11-04T00:00:00+00:00https://gfrison.com/2019/11/04/automated-negotiation<p>How could it be if you have a system that scales your sales force, encodes the knowledge of your sales representatives, contracts deals and negotiates the best terms and conditions on your behalf, with no effort and no waste of time?</p>
<p><em>I wrote it on https://cxlabs.sap.com/2019/11/04/automated-negotiation-with-zefiro/</em></p>
<p>Negotiation is ubiquitous in our lives. We negotiate with friends, employers, partners and of course car dealers, the business domain of choice for Zefiro.</p>
<blockquote>
<p>Negotiation is a method for finding an agreement in the presence of conflicting preferences and goals.</p>
</blockquote>
<p>What could be the preferences and goals in the sphere of commerce? For the buyer - let’s give a name, John - the objective values are the price and the qualities of the product. Other more subjective values are the emotional and social ones. The seller, on the other hand, is concerned on reaching targets and attaining them efficiently by reducing sales cycle time. But how differently their objective might appear, both of them want to close the deal, and get it under satisfactory terms.</p>
<p>Let’s come back to John. He’s looking for a car and starts his purchase journey when he’s searching for information on the internet. He reads reviews, customer’s feedback. The opinions from his friends are surveyed for making up his mind about what is suitable for him. Then it’s time to pay a visit to the car dealer, and that’s where the game with seller begins.</p>
<p><em>“…isn’t it amazing? I think we also have the ‘midnight blue’ color off-the-shelf. It arrived just yesterday.”</em></p>
<p><em>“I agree, blue is much better! Even better with 18 inches wheels.”</em></p>
<p><em>“I’m quite sure we have it, but I need to check in my computer, would you come?”</em></p>
<p><em>“Yes… but.. roughly, how much it costs?”</em></p>
<p><em>“Well, what monthly payment best fits with you? Let’s sit down in the office…”</em></p>
<p>The seller grasps everything is important to know about John. The needs, the expectations, the preferences and values will shape the desired car the seller will recommend. The perception of John’s financial status might exclude some product offerings in favor of others. The seller observes John’s expertise about cars and his ability in negotiating and he adapts the selling approach accordingly. The experienced salespeople leverages any tiny detail for driving the purchase process towards their goals.</p>
<p>Let’s assume a different scenario. John goes to the car dealer on Saturday afternoon and there are already ten customers, but there are only three salespersons. He roams in the showroom waiting for a seller…
You might think at this point, what if we can give John a tablet with an application that carries on the preliminary phase with a digitized version of the car dealer’s salespeople?</p>
<p>What might be the characteristics of such software application?</p>
<h2 id="real-time-learning">Real-time Learning</h2>
<p>Zefiro will learn at every interaction something more about John. His requirements, intentions, his car usage, his status and even whether he has kids or not. Do you know the “Guess the Celebrity Game” in which you try to get your team to guess which celebrity you are? Zefiro plays the celebrity seeker role and capitalizes all clues John will communicate directly and even indirectly. If tomorrow John would come back again in the showroom, Zefiro - and consequently the marketing and the sales department - already knows a bit of John.
When employed at the car dealer, Zefiro has acquired all relevant expertise for conducting the negotiation. It’s prior knowledge comes from marketing research, sales analytics and everything which distinguish that business. Several sources will feed its knowledge graph.</p>
<h2 id="adaptive-selling">Adaptive Selling</h2>
<p>John, exactly like you, is different from anyone else. Zefiro will adapt its selling strategy according to John’s personality. Adaptive Selling is a flexible approach to the negotiation. The selling method will be the optimal one for that type of customer and for that sales situation. Is that worthy? In case of wrong strategy the probability of purchase might drop down to -60% and the price concessions might rise up to +30%. Definitely, yes.</p>
<h2 id="self-learned-decision-making">Self-learned Decision Making</h2>
<p>There are many tools for helping salespeople to do their job. Analytics tools come in rescue of the sales process. Correct me if I’m wrong, but all interactions with clients, even the back and forward of EMails are manually responded. That’s a bottleneck Zefiro doesn’t have. Zefiro plays against many “virtual Johns” a considerable number of simulations, and it learns. It optimizes the selling process in its favor. It takes all decision steps, automatically.</p>
<h2 id="automated-argumentation">Automated Argumentation</h2>
<p>The system will stimulate, persuade and convince John that the car dealer proposal is the most convenient, advantageous deal he can’t get from another competitor in the area.</p>
<p>Zefiro could be able to meet expectations of sales even though it will be used just to relieve the sales force of the burden to serve the first line of contact, which is well known to have low conversion rate, and therefore very time wasting. The articulation of the system and the resolution of its actions might scale to a high level of negotiation management, and if properly expanded, the system would be able to handle complicated processes thanks to its unbounded optimization capabilities.</p>
<p>I didn’t write about other business cases so far. Are you curious to know how it can fit with yours?
<a href="mailto:giancarlo.frison@sap.com">Drop me a line</a>, and let’s sit down in the office…</p>Giancarlo FrisonHow could it be if you have a system that scales your sales force, encodes the knowledge of your sales representatives, contracts deals and negotiates the best terms and conditions on your behalf, with no effort and no waste of time?Machine Learning and BigData will make Precision Medicine the future of Healthcare2019-09-27T00:00:00+00:002019-09-27T00:00:00+00:00https://gfrison.com/2019/09/27/precision-medicine-healthcare<p>According to an <a href="https://www.bmj.com/content/353/bmj.i2139">article</a> published by the British Medical Journal in 2013, medical errors are the third cause of death in the US, and it is <a href="https://qualitysafety.bmj.com/content/22/Suppl_2/ii21">prudent assumption</a> that wrong diagnosis would have a strong incidence in errors that can be harmful to patients. 10-15% of diagnoses are wrong and 35% of adults experienced a medical mistake in the past 5 years involving themselves, their family, or friends. Without considering extreme consequences, the chance to receive a false diagnosis is almost certain along the entire span of life.</p>
<p><em>originally posted on https://cxlabs.sap.com/2019/06/18/data-analytics-for-precision-medicine/</em></p>
<center><img src="https://gfrison.com/assets/causes-of-death.png" /></center>
<p>It’s worth taking a moment to consider why these errors happen in the first place. There are more than 12.000 diseases according to the World Health Organization. Never heard of foreign accent syndrome or alien hand syndrome? Many doctors wouldn’t have either, they’re just two of the many new diseases that are constantly emerging. Among the causes of mistakes, there are the variability of how the disease might manifest. There are doctor variables such as expertise. We know also that errors occur either as a result of <a href="https://www.amjmed.com/article/S0002-9343%2808%2900040-5/fulltext">physicians’ overconfidence</a>. In the current era of machine learning and big data would be attainable to offer decision-support tools to fix the faulty synthesis of information? May as well be inspiring to look on how the automotive industry adopt those technologies for car maintenance.</p>
<p>Modern cars have many features for ensuring assistance and safety to drivers and passengers. The vehicles’ onboard computer make thousands of decisions based on hundreds of sensors, that among their purposes, perform diagnosis by monitoring internal signals for evidence of faults. Terabytes of data are processed under the hood for real-time analytics, and not just for audit the current state of the car, but also for suggesting the best treatment before real problems occur. In the long term, this can reduce the cost burden of maintenance. Could also predictive diagnostic save human lives by detecting diseases early and precisely even before they appear?</p>
<center><img src="https://gfrison.com/assets/snyderome.jpg" /></center>
<p>This what Prof. Michael Snyder <a href="https://www.ncbi.nlm.nih.gov/pubmed/22424236">performed</a> to himself, when he combined high-throughput technologies during 14 months of screenings. Why standard medical evaluations examine only a handful biomedical markers when the technology (in the far 2013) measured up to 40.000 elements and potentially learn a lot more? His paper reports more than 3 billion measurements (50TB of data) over infected and healthy states. Doing so, it is not only possible to identify physiological statuses that deviate from the healthy state but also do it long before the symptoms appear. Exactly like in your car’s monitoring system.</p>
<p>The concept of precision medicine (PM) hinges on one key premise: most diagnostics are designed for the “average patient” as a “one-size-fits-all” approach. But there is no such “average patient” and thus most treatments will be successful for some patients but not for others. Instead, therapies will be shaped to classes of patients on the basis of the differences in people’s genes, molecular information, clinical data, diet, environment, lifestyle and other biomarkers. Prof. Snyder felt that genomics “saved his life” because he found out about the diabetes early and he was able to make lifestyle changes, even though his health insurance premium became “prohibitively expensive”. However, he says the trade-off was worth it.</p>
<p>Precision medicine has emerged as a computationally approach to interpret the big data of pools of biological molecules (omics) and facilitate their application in healthcare. But doesn’t come without critics. One of them regards costs. Complete genomic sequencing became faster and less expensive with the introduction of new methods. However, reimbursement from insurance companies for these targeted screenings is likely to be still an issue. One of the most imposing barriers of PM may be the willingness and motivation of doctors to incorporate routine genomic testing into their daily workflows while the clinical benefits have not been definitively established.</p>
<p>Algorithmically, there is a shift to using informatics methods such as neural networks and advanced aggregative techniques to model complex relationships among patients within gene and biomolecules signatures to facilitate this process.
In the survey <a href="https://go.oracle.com/LP=66444?elqCampaignId=95819">Trends in Precision Medicine Adoption</a> made by GenomeWeb and sponsored by Oracle, the majority of respondents believe PM brings competitive advantages to their organizations while one of the most significant obstacles in pursuing PM is the lack of the information technology infrastructure to support the initiative.
Interpreting the amount of data collected by medical screenings and also by wearable devices (the so-called Digital Biomarkers) for multiple individuals would cause a structural bottleneck, that could be alleviated by a correspondent shift towards streaming and big data processing, a challenge that could be transformed into opportunity by first-class information providers.</p>Giancarlo FrisonAccording to an article published by the British Medical Journal in 2013, medical errors are the third cause of death in the US, and it is prudent assumption that wrong diagnosis would have a strong incidence in errors that can be harmful to patients. 10-15% of diagnoses are wrong and 35% of adults experienced a medical mistake in the past 5 years involving themselves, their family, or friends. Without considering extreme consequences, the chance to receive a false diagnosis is almost certain along the entire span of life.Photography Event in Bassano del Grappa2019-09-08T00:00:00+00:002019-09-08T00:00:00+00:00https://gfrison.com/2019/09/08/photo-exposition-bassano-fotografia<p>I’ve been since long time fascinated by photography and its power to capture my fantasy and attention. I was pretty delighted
to attend artistic initiatives, that by crossing the boundaries of museums and ateliers, they can reach unforeseen spectators in very unconventional settings.
The “FuoriSalone” in Milano, for example, is a majestic example of that. Arts, design and springing creativity flourish in areas
where people are used to see anything but design masterpieces. Similarly, in one of the cutest towns on the foot of alps, Bassano del Grappa,
is going to <a href="http://www.bassanofotografia.it/">take place</a> a photography event that will cover the historic center with splendid images.</p>
<p>I’m honored to give a small contribution, by exposing some pictures of mine in a couple of remarkable sites of the town: the main square (Piazza Liberta’) with its solar clock
and Viale dei Martiri, where visitors can admire the view of the mountains and the medieval towers.</p>
<h2 id="san-marco-caffe">San Marco Caffe’</h2>
<p>The Caffe’ is <a href="https://goo.gl/maps/fEBrUCyXhznyANbB6">located in the main square</a>, and visitors can see a glimpse of pictures even from the Piazza Liberta’.</p>
<blockquote>
<p>Our curse is that we are forced to interpret life as a sequence of events and that when we can’t figure out what our particular story is we feel lost somehow.</p>
</blockquote>
<p>The quote reflects somehow the experiment I performed during the Faschingsdienstag (Carnival Tuesday) in München:</p>
<center><img title="Giancarlo Frison - Pin-ups" src="https://gfrison.com/assets/marienplatz3.jpg" /></center>
<center><img title="Giancarlo Frison - Skates" src="https://gfrison.com/assets/skate.jpg" /></center>
<h2 id="les-amis">Les Amis</h2>
<p>Along the Viale dei Martiri, a <a href="https://goo.gl/maps/VuQTRVZxU3WmVoKUA">romantic shop</a> with clothing for woman is hosting a couple of photos taken during an open
concert in the Nymphemburg Park (München), <em>Serenade im Park</em>, few summers ago:</p>
<center><img title="Giancarlo Frison - Skates" src="https://gfrison.com/assets/serenade-im-park1.jpg" /></center>
<center><img title="Giancarlo Frison - Skates" src="https://gfrison.com/assets/serenade-impark2.jpg" /></center>
<p>Any preference on those pictures?</p>Giancarlo FrisonI’ve been since long time fascinated by photography and its power to capture my fantasy and attention. I was pretty delighted to attend artistic initiatives, that by crossing the boundaries of museums and ateliers, they can reach unforeseen spectators in very unconventional settings. The “FuoriSalone” in Milano, for example, is a majestic example of that. Arts, design and springing creativity flourish in areas where people are used to see anything but design masterpieces. Similarly, in one of the cutest towns on the foot of alps, Bassano del Grappa, is going to take place a photography event that will cover the historic center with splendid images.Dynamic Programming2019-06-18T00:00:00+00:002019-06-18T00:00:00+00:00https://gfrison.com/2019/06/18/dynamic-programming<p>The term <em>dynamic programming</em> has a curious origin.
When Richard Bellman late in the 1940s were seeking for a viral definition of his method, his boss was apparently not very inclined on science and in particular on mathematical research, the activities that Bellman was deep into when he formulate his famous <a href="https://en.wikipedia.org/wiki/Bellman_equation">equation</a>.
Likewise in marketing campaigns, where names and terms are carefully selected for attention teasing, Bellman coined the definition that combines the multi-staging and time-varying of <em>‘dynamic’</em> with the optimization archetyping of <em>‘programming’</em>, coincidentally induced by the fame of Dantzig’s <em>linear programming</em> for mathematical optimization.</p>
<figure>
<a href="https://www.flickr.com/photos/abbiateci64/32615368572"><img title="Umberto Boccioni - Rissa in Galleria(1910) by Maurizio Abbiateci (CCBY2)" src="https://gfrison.com/assets/rissa-galleria.jpg" /></a>
<figcaption>Umberto Boccioni - Rissa in Galleria(1910) by Maurizio Abbiateci (CCBY2)</figcaption>
</figure>
<p><em>DP</em> describes problems that involve dynamic processes for finding the best decision one after another. The backbone idea rolls around the splitting of a problem till its atomic parts are identified, then trimming those parts with a objective function. Basically, the core concept of <em>recursion</em>. Considering for example the Fibonacci algorithm:</p>
<center><img title="Fibonacci" src="https://gfrison.com/assets/fib-tree.png" /></center>
<p>The node <code class="language-plaintext highlighter-rouge">fib(2)</code> appear twice in the tree, therefore Fibonacci unveils an <em>overlapping</em> structure. It is eligible to be an <em>DP</em> algorithm. Not all recursive algorithms are inherently overlapping, look at <em>binary tree search</em>, <em>merge</em> and <em>quick sort</em>, for example. They don’t manifest the property of traversing smaller chunk of data more then once, hence they cannot join Dynamic Programming’s family. When the sub-problems are repeated in the problem as whole, and their values are evaluated all over again unless a caching mechanism (<em>memoization</em>) could be displaced for preventing such inefficiency.</p>
<p>Bellman also conceived the <em>principle of optimality</em> according to which an optimal policy should always hands out the optimal decision from any state or action previously done. However the chessboard pieces’ are laid, or however pedestrian are crossing the road or waiting on the platform, the agent will move pieces properly in case of chess and it will skirt traversing people in case of self-driving car. The automatic actor will always follow the best strategy from the first step and thereafter.
This is for introducing another property of <em>DP</em> methods, which is the <em>optimal structure</em>.</p>
<center><img title="shortest path problem" src="https://gfrison.com/assets/shortest-path.png" /></center>
<p>When a courier has to deliver a packet from <script type="math/tex">A</script> to the destination <script type="math/tex">F</script>, he will catch the shortest way highlighted in blue. The optimal decision for completing the journey <script type="math/tex">A \rightarrow F</script>, in the shortest path problem, includes necessarily the optimal solution <script type="math/tex">C \rightarrow F</script>. As you may have noticed, The recursion sweats also out of this property where nested sub-problems are always optimal on their way to the final target. The shortest path problem present the <em>optimal sub-structure</em>, which claims that an optimal solution of a problem includes necessarily the optimal solutions of its sub-problems. Not all recursive problems are optimal even in their encapsulated structure, and the counterfactual is given by slightly different dilemma, the longest path problem. The longest way <script type="math/tex">A \rightarrow F</script> includes the node <script type="math/tex">C</script>, but if we choose to start from <script type="math/tex">C</script>, the path is not included in the way starting from <script type="math/tex">A</script>.</p>
<h2 id="dynamic-programming-in-reinforcement-learning">Dynamic programming in reinforcement learning</h2>
<p>The first time <em>DP</em> and <em>RL</em> were mentioned together was by Minsky in the 1961 and it took form of the Bellman equation, because <em>RL</em> problems have expose usually an overlapping and optimal structure, ideal for being solved by <em>DP</em>.</p>
<h3 id="markov-decision-process">Markov decision process</h3>
<p><a href="/2018/12/26/first-steps-reiforcement-learning/">Reinforcement learning</a> is a class of methods for determining the optimal policy an agent should apply for maximize its return in a given environment. The entities with a role in <em>RL</em> are the state <script type="math/tex">S</script>, the mutable conditions an agent experience in an environment, the action <script type="math/tex">A</script> the agent execute in a particular state, and the reward <script type="math/tex">R</script>, if any, which score the goodness of action taken in a state. These are the pillars of the <a href="https://en.wikipedia.org/wiki/Markov_decision_process">markov decision process</a>, the framework that formalizes the policy <script type="math/tex">π</script> as a sequence of steps <script type="math/tex">(S_t, A_t, R_t)</script>, aggregated in episodes.</p>
<center><img title="MDP" src="https://gfrison.com/assets/mdp-states.png" /></center>
<p>The <em>policy</em> is a mapping from states and related actions, <em>RL</em> tells us how the agent’s policy changes as result of the experience.
For finding good policies, we need to estimate how good it is, in terms of future rewards, to be in a particular state. Value functions <script type="math/tex">v_\pi(s)</script> define the expected return when starting from a given state <script type="math/tex">s</script> and following <script type="math/tex">\pi</script> thereafter. their fundamental property is that they satisfy recursive relationships similar to what we already have seen for dynamic programming. Hence, the idea of <em>DP</em> in <em>RL</em> is the use of value functions to organize the search for good policies.</p>
<h3 id="the-gridworld-example">The Gridworld example</h3>
<table>
<thead>
<tr>
<th style="text-align: center"><img src="https://gfrison.com/assets/gridworld.png" alt="GridWorld" /></th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align: center"><em>Problem: Find all trajectories towards the flags from any cell</em></td>
</tr>
</tbody>
</table>
<p>Among many examples I may show for explaining what does <em>DP</em> stands for, this very simple <em>GridWorld</em> conveys the idea of an agent displaced in an 2D environment. The reward drives the agent to move toward the flagged corners regardless the position where the agent is located.
How the agent could learn the optimal ways? I give you an hint, let’s start from the end, the flagged boxes.
In that position you don’t have anymore rewards to gather, the task is brilliantly completed, so we assume the terminal’s state value is is equal to <em>zero</em>.</p>
<table>
<thead>
<tr>
<th style="text-align: center"><img src="https://gfrison.com/assets/mdp-t1.png" alt="GridWorld" /></th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align: center"><em>Iteration 1. Terminal state value is zero</em></td>
</tr>
</tbody>
</table>
<p>From end position, step back and look around. Among the actions you can do, the move that catapult you into the terminal state is more rewarding than others (<em>zero</em> instead of <em>-1</em>).
Congratulations! You already have solved a piece of the puzzle. It is just the last action, you still have to solve the rest of the grid. In that position, as in any other cell of the grid, you need to assign a value to the cell where you are in that particular moment, in order to pave the way to the final goal. That value <script type="math/tex">Q</script> indicates how good it is to be there. Not all cells carry the same value, some are more valuable than others. The agent, when choosing the next move, will move to the cell with highest value, among the surroundings. The value would not be just the reward you get in there (<em>-1</em>), but the reward plus the average of the values of the cells proximate to you.
<script type="math/tex">Q_t = r_t + \sum_{n=t+1}{Q_n}</script></p>
<table>
<thead>
<tr>
<th style="text-align: center"><img src="https://gfrison.com/assets/mdp-t2.png" alt="GridWorld" /></th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align: center"><em>Iteration 2: set the terminal state value to _zero_</em></td>
</tr>
</tbody>
</table>
<p>Going backward till the initial position, you see all rewards on the way to the terminal state waiting for you to being picked up. This is the value of your state.
You are in the middle of the trajectory towards the end, and the value of your state is equal of the reward in that state plus the sum of the discounted rewards thereafter.
Clearly, the agent moves toward the most promising among the surrounding cells, once it has realized their value. Let’s keep going.
The grid starts to unveil the optimal trajectories by just going backward and evaluating what could be the best move, taking the cell with highest value.
A pattern is identifiable, something that programmers knows well, the <em>recursion</em>, and simplified as:</p>
<script type="math/tex; mode=display">Q_t=r_t+argmax(Q_{t+1})</script>Giancarlo FrisonThe term dynamic programming has a curious origin. When Richard Bellman late in the 1940s were seeking for a viral definition of his method, his boss was apparently not very inclined on science and in particular on mathematical research, the activities that Bellman was deep into when he formulate his famous equation. Likewise in marketing campaigns, where names and terms are carefully selected for attention teasing, Bellman coined the definition that combines the multi-staging and time-varying of ‘dynamic’ with the optimization archetyping of ‘programming’, coincidentally induced by the fame of Dantzig’s linear programming for mathematical optimization. Umberto Boccioni - Rissa in Galleria(1910) by Maurizio Abbiateci (CCBY2)