I may not be an expert but I have following observations:
First of all, awesome & clear explanation of problem from your side.
As rightly suggested by Peter, there are two points of contacts; and that should create a moment which is countered by the fixed support.
Considering this as cantilever problem (sorry not a block on a wedge problem, although you did solve it correctly; not useful in this case) When we do force equilibrium, we get reaction Fz = mg which it should show and no initial force in Y direction will give reaction Fy = 0 N as shown by software.
But because there is two point contact, it should give a moment as shown in poorly drawn art work above but there should be a reaction moment too at fixed support in my opinion (and not force in Y direction).
1) Why is the pin completely undeformed? Is there an error or do you think it could be physically accurate? If it is an error how can I fix it? Using bounded contact instead of friction?
--> Bonded contact wont give realistic result (It should ideally slip from the conical surface). Now, what I think reason for no deformation in conical pin is that, You either made it rigid body (which is not the case here because you cannot cut rigid body using section plane tool in result) or maybe because the length of the pin is considerably short compared to distance of point mass to conical pin or the deflection is very less compared to other deflection, maybe try scaling the result, adding (+) more color pallet in the result and check ?
2) How could I improve the mesh of the target body? The automatic methods are not applicable so I need an alternative strategy
--> Check if it is a sweepable body or not (mesh--> show -- show sweepable bodies), if it is, go for sweep method with manual target and contact faces if not I would suggest method of hex dom) for better mesh.
Or manually, reduce the element size if you want to. I don't understand "automatic methods are not applicable" statement.
Thanks Chinmay