Not as far as I know, from the help file - "Only files that have objects in Model Space can be added as Xref's." I would think it could become confusing if multiple x-refs are added, each with various paperspaces, layers, styles etc. Especially if like me, one uses the same viewport names, layer names etc in multiple drawings.
As a side note - one can 'add' files with objects, paper-spaces, viewports etc. into a drawing, by using the Extract From, BUT one would lose any advantage of x-refs, in that the drawing would not update if the original is altered.