GATE EC 2022 Set 1 — Question 29

The question asks to identify the statements that are necessarily true regarding the entropy H(X) of a discrete random variable $X$ taking $K$ distinct real values.Option (A): $H(X) \leq \log_2 K$ bits
This is a fundamental property of entropy. The maximum entropy for a discrete random variable with $K$ possible outcomes occurs when all outcomes are equally probable (i.e., a uniform distribution). In this case, $H(X) = \sum_{i=1}^K P(x_i) \log_2 \frac{1}{P(x_i)} = \sum_{i=1}^K \frac{1}{K} \log_2 K = K \cdot \frac{1}{K} \log_2 K = \log_2 K$. For any other probability distribution, the entropy will be less than $\log_2 K$. Thus, $H(X) \leq \log_2 K$ is always true. This statement is correct.Option (B): $H(X) \leq H(2X)$
Let $Y = 2X$. If $X$ takes $K$ distinct values $x_1, x_2, \dots, x_K$, then $2X$ takes $K$ distinct values $2x_1, 2x_2, \dots, 2x_K$. The mapping $f(x) = 2x$ is a one-to-one function. For any one-to-one deterministic function $Y = f(X)$, the entropy remains unchanged, i.e., $H(Y) = H(X)$. Therefore, $H(2X) = H(X)$. The statement $H(X) \leq H(2X)$ is true because $H(X) = H(2X)$. This statement is correct.Option (C): $H(X) \leq H(X^2)$
Let $Y = X^2$. The mapping $f(x) = x^2$ is not necessarily a one-to-one function. For example, if $X$ can take values $-1$ and $1$ with $P(X=-1) = 0.5$ and $P(X=1) = 0.5$, then $H(X) = - (0.5 \log_2 0.5 + 0.5 \log_2 0.5) = - (0.5(-1) + 0.5(-1)) = 1$ bit. However, $X^2$ will only take the value $1$ (since $(-1)^2 = 1$ and $1^2 = 1$). So, $P(X^2=1) = 1$, and $H(X^2) = - (1 \log_2 1) = 0$ bits. In this case, $H(X) = 1 > H(X^2) = 0$. Therefore, $H(X) \leq H(X^2)$ is not necessarily true. This statement is incorrect.Option (D): $H(X) \leq H(2^X)$
Let $Y = 2^X$. The mapping $f(x) = 2^x$ is a one-to-one function for real values of $x$. If $X$ takes $K$ distinct values, then $2^X$ will also take $K$ distinct values. Similar to option (B), for a one-to-one deterministic function, the entropy remains unchanged. Therefore, $H(2^X) = H(X)$. The statement $H(X) \leq H(2^X)$ is true because $H(X) = H(2^X)$. This statement is correct.The final answer is $\boxed{A, B, D}$