Given $\vec{r}=3\hat{i}-2\hat{j}+6\hat{k}$

First, we find $\vec{r}\times\hat{j}$:

$\vec{r}\times\hat{j} = (3\hat{i}-2\hat{j}+6\hat{k})\times\hat{j} = 3(\hat{i}\times\hat{j}) - 2(\hat{j}\times\hat{j}) + 6(\hat{k}\times\hat{j}) = 3\hat{k} - 0 - 6\hat{i} = -6\hat{i} + 3\hat{k}$

Next, we find $\vec{r}\times\hat{k}$:

$\vec{r}\times\hat{k} = (3\hat{i}-2\hat{j}+6\hat{k})\times\hat{k} = 3(\hat{i}\times\hat{k}) - 2(\hat{j}\times\hat{k}) + 6(\hat{k}\times\hat{k}) = -3\hat{j} - 2\hat{i} + 0 = -2\hat{i} - 3\hat{j}$

Now, we find $(\vec{r}\times\hat{j})\cdot(\vec{r}\times\hat{k})$:

$(\vec{r}\times\hat{j})\cdot(\vec{r}\times\hat{k}) = (-6\hat{i} + 3\hat{k})\cdot(-2\hat{i} - 3\hat{j}) = (-6)(-2) + (0)(-3) + (3)(0) = 12 + 0 + 0 = 12$

Finally, we find $(\vec{r}\times\hat{j})\cdot(\vec{r}\times\hat{k}) - 12$:

$12 - 12 = 0$